cmd.exe on WIndows - problem with displaying some Unicode characters

[Wiktor]

Hi,

as OO programming exercise, I'm trying to port to Python one of my favorite
game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
version on C64

and here's
video from modern rewritten (for Atari emulators) version: Kolony 2106

- you get the idea? ;-)).

OO Design is one thing, but I want to make it look as near as possible to
the original (those windows-like menus in console window). I tried to use
'standard' Unicode characters (I can see that most of my Windows monospaced
fonts have them) to draw frame around menu. Something like this:

????????????????
? Construction ?
? Production ?
? Research ?
? Exploration ?
????????????????
? Next turn ?
????????????????

(I like the look of double lines on right and at the bottom)
But when I try to print those characters, I get an error:

| Traceback (most recent call last):
| File "E:\Moje dokumenty\python\kolony\menu.py", line 14, in <module>
| """
| File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
| return codecs.charmap_encode(input,self.errors,encoding_map)[0]
| UnicodeEncodeError: 'charmap' codec can't encode character '\u2556' in position 1
| 6: character maps to <undefined>

Now I know what that means. Code page that my cmd.exe is using (852)
doesn't have "?", "?", "?" and "?" symbols. Changing code page to Unicode
(65001) doesn't really help, because all is messed up:
????????????????
? Construction ?
? Production ?
? Research ?
? Exploration ?
????????????????
? Next turn ?
????????????????
???????????
? Next turn ?
????????????????
???????????????
????????
??
(I believe that's cmd.exe bug with Unicode support, not Python fault)


Before I drop entirely this idea of using double lines on right and bottom
edges, and make it look like this
????????????????
? Construction ?
?--------------?
? Next turn ?
????????????????
I have to ask - is there a way to make that original concept work? I know,
that CP437 has symbols "?", "?" and "?", but does not have polish letters -
and I need to display them too.
I also know, that cmd.exe can display those Unicode characters (by
copy/paste them in command line or by listing filenames containing that
characters), no matter what CP is set. How does it manage to do it? Can I
exploit that writing my Python program?
Wiktor

[Chris Angelico]

Post by Wiktor
I have to ask - is there a way to make that original concept work? I know,
that CP437 has symbols "?", "?" and "?", but does not have polish letters -
and I need to display them too.

Yeah, that's exactly the problem with codepages :)

The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.

One option would be to render the whole thing graphically, abandoning
cmd.exe altogether. That would be how a lot of telnet and SSH clients
will do the work. Get a proper Unicode-supporting toolkit (Tkinter has
issues with astral characters too, AIUI), and yes, you'll have to do a
lot of work yourself. Or maybe, grab an actual telnet client, and
write this as a socket server. I'd be delighted to help you with that
option - I'm a MUDder and have spent innumerable dev hours on telnet
clients!

ChrisA

[Andrew Berg]

Post by Chris Angelico
The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.

That is not quite true. The terminal has these issues, not the shell. Using
cp65001 does make Unicode in a Windows terminal possible, but using a better
terminal[1] makes it almost perfect (my experience has been that input can be
difficult, but output works well). I personally have used an IRC bot written in
Python with logging output containing Unicode characters that display just fine
(both locally and over SSH).

[1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/

[Mark Lawrence]

Post by Andrew Berg

Post by Chris Angelico
The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.

That is not quite true. The terminal has these issues, not the shell. Using
cp65001 does make Unicode in a Windows terminal possible, but using a better
terminal[1] makes it almost perfect (my experience has been that input can be
difficult, but output works well). I personally have used an IRC bot written in
Python with logging output containing Unicode characters that display just fine
(both locally and over SSH).
[1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/

*facepalm* forgot all about ConEmu, but then I only use it on a daily
basis :)

[Chris Angelico]

On Mon, Aug 4, 2014 at 9:25 AM, Andrew Berg

Post by Andrew Berg

Post by Chris Angelico
The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.

That is not quite true. The terminal has these issues, not the shell. Using
cp65001 does make Unicode in a Windows terminal possible, but using a better
terminal[1] makes it almost perfect (my experience has been that input can be
difficult, but output works well). I personally have used an IRC bot written in
Python with logging output containing Unicode characters that display just fine
(both locally and over SSH).
[1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/

Sorry, yeah, my terminology was sloppy. It's not technically cmd.exe,
but the default console.

I just played around with a CP-437 decode of everything 128-255,
rendered in various different fonts, all using my MUD client on
Windows. (For what it's worth, it renders using GTK2 and Pango. But I
suspect this is more a font issue than a display engine one.) Most
fonts had those little boxes-with-numbers for most of the line drawing
characters, but Lucida Sans Unicode, Meiryo, and Tahoma all managed to
display all the characters. However, I wasn't able to visually
distinguish between the single-line and double-line characters, so you
may want to play around with it a bit more.

Your simplest solution may still be to abandon those double lines and
just go with single.

ChrisA

[Chris Angelico]

Post by Chris Angelico
I just played around with a CP-437 decode of everything 128-255,
rendered in various different fonts, all using my MUD client on
Windows. (For what it's worth, it renders using GTK2 and Pango. But I
suspect this is more a font issue than a display engine one.) Most
fonts had those little boxes-with-numbers for most of the line drawing
characters, but Lucida Sans Unicode, Meiryo, and Tahoma all managed to
display all the characters. However, I wasn't able to visually
distinguish between the single-line and double-line characters, so you
may want to play around with it a bit more.

Hmm. Actually... I'd mucked up my CP-437 decode. For instance, I'd
represented 0xC8 as U+251D BOX DRAWINGS VERTICAL LIGHT AND RIGHT
HEAVY, where it's actually U+255A BOX DRAWINGS DOUBLE UP AND RIGHT.
When I get the right characters, the default fonts on Windows work
just fine. (There's still the issue that the supposedly "HEAVY" lines
are coming out with the exact same thickness as the "LIGHT" lines, but
that's unlikely to bother you as you don't need those characters.)

So, conclusion: If you do the drawing yourself, all those characters
are fine. Go ahead and use them!

ChrisA

[Glenn Linderman]

Post by Andrew Berg

Post by Chris Angelico
The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.

That is not quite true. The terminal has these issues, not the shell. Using
cp65001 does make Unicode in a Windows terminal possible, but using a better
terminal[1] makes it almost perfect (my experience has been that input can be
difficult, but output works well). I personally have used an IRC bot written in
Python with logging output containing Unicode characters that display just fine
(both locally and over SSH).
[1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/

I will be reading more about conemu, thanks for the reference.

http://bugs.python.org/issue1602 describes 7 years worth of discussion
of the problems with the console/terminal used by default by cmd.exe and
other Windows command line programs, versus Python.

The recent insights in the last couple weeks have given me hope that
Python might be able to be fixed to work properly with the default
Windows console at long last... at least for non-astral characters (I'm
not sure whether or not the Windows console supports non-BMP characters).

For this OP problem, it is mostly a matter of finding a fixed-width font
that supports the box drawing characters and the Polish characters that
are desired. Lucida Console has a fair repertoire, and Consolas has a
fair repertoire, in the fixed-width font arena. There may be others,
documented on Polish language web sites that I wouldn't know about, and
I don't know enough Polish to be sure those I mentioned suffice.

And then, the workarounds mentioned in the above-referenced bug or on
the GitHub or PyPi sites mentioned should provide any needed additional
solutions... and hopefully something along this line finally integrated
into Python so that it can finally be said that Python supports Unicode
properly on Windows (or at least as properly as Windows allows... but it
is pretty clear that Windows supports Unicode, even for the console,
using different APIs that Python is presently using, and that mismatch
between APIs is really the source of the problems with using Unicode in
Python on Windows).

Glenn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140803/0661683b/attachment.html>

[Glenn Linderman]

Post by Glenn Linderman

Post by Andrew Berg

Post by Chris Angelico
The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.

That is not quite true. The terminal has these issues, not the shell. Using
cp65001 does make Unicode in a Windows terminal possible, but using a better
terminal[1] makes it almost perfect (my experience has been that input can be
difficult, but output works well). I personally have used an IRC bot written in
Python with logging output containing Unicode characters that display just fine
(both locally and over SSH).
[1] I recommend ConEmu:https://code.google.com/p/conemu-maximus5/

I will be reading more about conemu, thanks for the reference.

Having read a bit about ConEmu, it seems that it is a "pretty face"
built on top of Windows Console, by screen scraping the real (but
hidden) Windows Console, and providing a number of interesting display
features and modes. So while it adds functionality to the Windows
Console interface, it doesn't seem like it is likely to fix bugs or
resolve issues with code pages, font selection, or Unicode character
repertoires, which are the issues of this thread and the bug I
referenced earlier.

Can anyone with ConEmu installed refute this interpretation of its
functionality?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140803/a350e699/attachment.html>

[Andrew Berg]

Having read a bit about ConEmu, it seems that it is a "pretty face" built on
top of Windows Console, by screen scraping the real (but hidden) Windows
Console, and providing a number of interesting display features and modes. So
while it adds functionality to the Windows Console interface, it doesn't seem
like it is likely to fix bugs or resolve issues with code pages, font
selection, or Unicode character repertoires, which are the issues of this
thread and the bug I referenced earlier.
Can anyone with ConEmu installed refute this interpretation of its functionality?

If you run cmd in it, you will still need to use cp65001. This is not necessary
for (or applicable to) other applications (such as a Python interpreter) run
directly. ConEmu can use any arbitrary font available on the system. As I have
said, I have been able to display Unicode output on it from an application
written in Python. No mojibake, no replacement characters, just the exact
characters one would expect.
I do not know the internals of ConEmu and how it interacts with conhost and
whatever else, but I have not found a need to since it has just worked for me.

[Glenn Linderman]

Post by Andrew Berg

Having read a bit about ConEmu, it seems that it is a "pretty face" built on
top of Windows Console, by screen scraping the real (but hidden) Windows
Console, and providing a number of interesting display features and modes. So
while it adds functionality to the Windows Console interface, it doesn't seem
like it is likely to fix bugs or resolve issues with code pages, font
selection, or Unicode character repertoires, which are the issues of this
thread and the bug I referenced earlier.
Can anyone with ConEmu installed refute this interpretation of its functionality?

If you run cmd in it, you will still need to use cp65001.

Or some workaround. The latest workaround in the issue I referenced does
not require cp65001 either, for output, at least.

Post by Andrew Berg
This is not necessary
for (or applicable to) other applications (such as a Python interpreter) run
directly.

How does one "directly run" another application using ConEmu? That
wasn't clear from what I found to read. It sounded like you run ConEmu,
run one or more shells within it, and launch programs from those shells?
And so it was also unclear if a program launched from some batch file
would have to have the batch file launched from ConEmu, also. Or does
ConEmu grab the execution association for batch files to make that work
more automatically?

Post by Andrew Berg
ConEmu can use any arbitrary font available on the system. As I have
said, I have been able to display Unicode output on it from an application
written in Python. No mojibake, no replacement characters, just the exact
characters one would expect.
I do not know the internals of ConEmu and how it interacts with conhost and
whatever else, but I have not found a need to since it has just worked for me.

So you may not know the internals of ConEmu, but I presume you know the
internals of your Python applications. What encodings do you use for
stdout for those applications? Do you set up the Python environment
variables that specify some particular encoding, in the ConEmu
environment (or does it)? Because the default Python IO encoding in
Windows seems to be obtained from the configured code page.

Of course the biggest problem with much free and open source software is
the documentation; I wasn't able to find specific answers to all my
questions by reading the ConEmu wiki. Maybe some of it would be clearer
if I installed it, and your "just worked" comment is certainly
encouraging me to "just try it", but while trying it may help me figure
it out, adding another package to separately install for my users gives
more complexity. See if you can push me over the edge :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140804/8e7547bb/attachment.html>

[Andrew Berg]

How does one "directly run" another application using ConEmu? That wasn't clear
from what I found to read. It sounded like you run ConEmu, run one or more
shells within it, and launch programs from those shells? And so it was also
unclear if a program launched from some batch file would have to have the batch
file launched from ConEmu, also. Or does ConEmu grab the execution association
for batch files to make that work more automatically?

When you open a new console, the dialog will ask you to supply a path to an
executable you want to run. Any arbitrary CLI application will work. I don't
understand your question about batch files. If you mean to ask if ConEmu will
snatch a console opened by executing a batch file outside of ConEmu, yes, it
can do that. I highly suggest actually using the program and doing some tests
of your own to see how it works.

Post by Andrew Berg
ConEmu can use any arbitrary font available on the system. As I have
said, I have been able to display Unicode output on it from an application
written in Python. No mojibake, no replacement characters, just the exact
characters one would expect.
I do not know the internals of ConEmu and how it interacts with conhost and
whatever else, but I have not found a need to since it has just worked for me.

So you may not know the internals of ConEmu, but I presume you know the
internals of your Python applications. What encodings do you use for stdout for
those applications? Do you set up the Python environment variables that specify
some particular encoding, in the ConEmu environment (or does it)? Because the
default Python IO encoding in Windows seems to be obtained from the configured
code page.

I use UTF-8. Open the Python interpreter directly in ConEmu and see what you get.

Of course the biggest problem with much free and open source software is the
documentation; I wasn't able to find specific answers to all my questions by
reading the ConEmu wiki. Maybe some of it would be clearer if I installed it,
and your "just worked" comment is certainly encouraging me to "just try it",
but while trying it may help me figure it out, adding another package to
separately install for my users gives more complexity. See if you can push me
over the edge :)

It certainly would make things much clearer if you were to actually use the
program. Documentation tends to assume (and reasonably so, IMO) that you intend
to do some hands-on learning.
I can give no advice on deploying this to your users other than to say ConEmu
works well as a tool for command line power users on Windows, but does not
provide much ROI when it is simply an implementation detail for a single
program. If you want to save your users the hassle, I would definitely
recommend a graphical environment. If I had realized that you intended your
application to be widely deployed, I would have simply recommended that from
the start.

On a side note, you would have run into similar issues on *nix systems where a
significant amount of your users would be using the "C" locale and have no idea
what it is, why it causes them problems, or how to change it.

[Glenn Linderman]

Post by Andrew Berg
If you want to save your users the hassle, I would definitely
recommend a graphical environment. If I had realized that you intended your
application to be widely deployed, I would have simply recommended that from
the start.

Graphical environments are good for some things, command line
environments are good for other things.

Unicode capability is beneficial in both.

Many of the software tools I create and distribute are command line
utilities, for people that use command lines anyway. The problems arise
when they are multilingual, and need to use diverse character repertoires.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140804/edaa0a98/attachment.html>

[Chris “Kwpolska” Warrick]

Post by Glenn Linderman
For this OP problem, it is mostly a matter of finding a fixed-width font
that supports the box drawing characters and the Polish characters that are
desired. Lucida Console has a fair repertoire, and Consolas has a fair
repertoire, in the fixed-width font arena. There may be others, documented
on Polish language web sites that I wouldn't know about, and I don't know
enough Polish to be sure those I mentioned suffice.

Not really. We haven?t had to play the ?custom fonts for our
language? game for quite some time. Consolas and Lucida Console work
just fine for Polish (though Consolas sometimes hiccups on capital
characters in some environments). The characters are:

?????????
?????????

Most fonts ? especially the ones included with modern OSes ? support
all 18 of them.

So, this is a non-issue. The real issue is Windows being an idiot
when it comes to CLI, which is oh so surprising to everyone who had to
work with it ? considering the choice of the outdated and quirky
cmd.exe interpreter, or PowerShell, which has ultra-verbose
human-unfriendly commands and works in the same cmd.exe window (there
is a thing named PowerShell ISE, which circumvents cmd.exe; I have no
idea whether all the Unicode issues apply to that, too)

[wxjmfauth]

Post by Chris Angelico
The best way to do it is to use the Unicode codepage, but cmd.exe just
plain has issues. There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.
That is not quite true. The terminal has these issues, not the shell. Using
cp65001 does make Unicode in a Windows terminal possible, but using a better
terminal[1] makes it almost perfect (my experience has been that input can be
difficult, but output works well). I personally have used an IRC bot written in
Python with logging output containing Unicode characters that display just fine
(both locally and over SSH).
[1] I recommend ConEmu: https://code.google.com/p/conemu-maximus5/
I will be reading more about conemu, thanks for the reference.
http://bugs.python.org/issue1602? describes 7 years worth of
discussion of the problems with the console/terminal used by default
by cmd.exe and other Windows command line programs, versus Python.
The recent insights in the last couple weeks have given me hope that
Python might be able to be fixed to work properly with the default
Windows console at long last... at least for non-astral characters
(I'm not sure whether or not the Windows console supports non-BMP
characters).
For this OP problem, it is mostly a matter of finding a fixed-width
font that supports the box drawing characters and the Polish
characters that are desired.? Lucida Console has a fair repertoire,
and Consolas has a fair repertoire, in the fixed-width font arena.
There may be others, documented on Polish language web sites that I
wouldn't know about, and I don't know enough Polish to be sure those
I mentioned suffice.
And then, the workarounds mentioned in the above-referenced bug or
on the GitHub or PyPi sites mentioned should provide any needed
additional solutions... and hopefully something along this line
finally integrated into Python so that it can finally be said that
Python supports Unicode properly on Windows (or at least as properly
as Windows allows... but it is pretty clear that Windows supports
Unicode, even for the console, using different APIs that Python is
presently using, and that mismatch between APIs is really the source
of the problems with using Unicode in Python on Windows).

1) A lot of confusion and imprecisions.

2) Unicode will never work properly because its handling
is wrong by design.

Post by Chris Angelico

me.centralwidget.shell.font().rawName()

'Consolas'

Post by Chris Angelico

me.centralwidget.shell.fontMetrics().width('?')

9

Post by Chris Angelico

me.centralwidget.shell.fontMetrics().width('?')

17

Post by Chris Angelico

me.centralwidget.shell.fontMetrics().width('\t')

80
(When I think I will thow away all this work...)

4) Already posted:

Fun with win_unicode_console

NB Modulo .notdef glyph in font.

D:\>D:\conuni\build\exe.win32-3.2\jmtest.exe
3.2.5 (default, May 15 2013, 23:06:03) [MSC v.1500 32 bit (Intel)]
Quelques caract?res: ?abc???????????
Loop: empty string => quit
?>for ascii users: abc
Votre entr?e ?tait : for ascii users: abc 20 caract?re(s)
?>abc????
Votre entr?e ?tait : abc???? 7 caract?re(s)
?>*?*\u20ac*
Votre entr?e ?tait : *?*?* 5 caract?re(s)
?>\\\
Votre entr?e ?tait : \\\ 3 caract?re(s)
?>\\u0066
Votre entr?e ?tait : \f 2 caract?re(s)
?>a???????z
Votre entr?e ?tait : a???????z 9 caract?re(s)
?>D:\jm\??????\Z?rich\?????\?dipe
Votre entr?e ?tait : D:\jm\??????\Z?rich\?????\?dipe 31 caract?re(s)
?>a\u123z
Wahrsheinlich falsches \uxxxx
?>r'\'
Votre entr?e ?tait : r'\' 4 caract?re(s)
?>?\u1234\u3456z
Votre entr?e ?tait : ???z 4 caract?re(s)
?>
Fin


Addendum: there is a "but".

5) I'm aware about the discussions on the subject, see 1).

jmf

[Mark Lawrence]

Post by Wiktor
Hi,
as OO programming exercise, I'm trying to port to Python one of my favorite
game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
version on C64 http://youtu.be/UFycYOp2cbE and here's
video from modern rewritten (for Atari emulators) version: Kolony 2106
http://youtu.be/eX20Qqqm5eg - you get the idea? ;-)).
OO Design is one thing, but I want to make it look as near as possible to
the original (those windows-like menus in console window). I tried to use
'standard' Unicode characters (I can see that most of my Windows monospaced
????????????????
? Construction ?
? Production ?
? Research ?
? Exploration ?
????????????????
? Next turn ?
????????????????
(I like the look of double lines on right and at the bottom)
| File "E:\Moje dokumenty\python\kolony\menu.py", line 14, in <module>
| """
| File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
| return codecs.charmap_encode(input,self.errors,encoding_map)[0]
| UnicodeEncodeError: 'charmap' codec can't encode character '\u2556' in position 1
| 6: character maps to <undefined>
Now I know what that means. Code page that my cmd.exe is using (852)
doesn't have "?", "?", "?" and "?" symbols. Changing code page to Unicode
????????????????
? Construction ?
? Production ?
? Research ?
? Exploration ?
????????????????
? Next turn ?
????????????????
???????????
? Next turn ?
????????????????
???????????????
????????
??
(I believe that's cmd.exe bug with Unicode support, not Python fault)
Before I drop entirely this idea of using double lines on right and bottom
edges, and make it look like this
????????????????
? Construction ?
?--------------?
? Next turn ?
????????????????
I have to ask - is there a way to make that original concept work? I know,
that CP437 has symbols "?", "?" and "?", but does not have polish letters -
and I need to display them too.
I also know, that cmd.exe can display those Unicode characters (by
copy/paste them in command line or by listing filenames containing that
characters), no matter what CP is set. How does it manage to do it? Can I
exploit that writing my Python program?
Wiktor

There are multiple known problems with cmd.exe and unicode. A solution
might be to use powershell, but windows being windows who really knows? :)

[Terry Reedy]

Post by Wiktor
Hi,
as OO programming exercise, I'm trying to port to Python one of my favorite
game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
version on C64 http://youtu.be/UFycYOp2cbE and here's

This appears to be an actual text screen, no graphics.

Post by Wiktor
video from modern rewritten (for Atari emulators) version: Kolony 2106
http://youtu.be/eX20Qqqm5eg - you get the idea? ;-)).

This appears to be text boxes on a graphics screen.

Post by Wiktor
OO Design is one thing, but I want to make it look as near as possible to
the original (those windows-like menus in console window).

Which original? the C64 or Atari. The important characteristic of both
is that both have multiple overlapping popup boxes. This means that
either you or a widget framework much keep track of stacking order and
how to restore what was hidden when a box goes away or is moved down in
the stacking order. I would not be surprised if the Atari had at least a
rudimentary widget framework.

Post by Wiktor
I tried to use
'standard' Unicode characters (I can see that most of my Windows monospaced
????????????????
? Construction ?
? Production ?
? Research ?
? Exploration ?
????????????????
? Next turn ?
????????????????
(I like the look of double lines on right and at the bottom)
| File "E:\Moje dokumenty\python\kolony\menu.py", line 14, in <module>
| """
| File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
| return codecs.charmap_encode(input,self.errors,encoding_map)[0]
| UnicodeEncodeError: 'charmap' codec can't encode character '\u2556' in position 1
| 6: character maps to <undefined>

You have two separate problems with running in the windows console.

1. The character issue. If you run a program to just print the above
from an Idle editor, so that the output is printed to the Idle shell,
there should be no problem.

Post by Wiktor

print('\u2556'*10)

??????????
But characters are not your real issue.

2. The random access issue. The MS console in normal use is like a
serial printer or terminal. Once a line is printed, it cannot be
changed. I looked at the video and the program randomly accesses a 24 or
25 line x 80 column screen, overprinting existing characters at will and
reversing black on white versus white of black at will. MSDOS screens
recognized standard ANSI screen control codes once the ANSI.SYS driver
was installed, which was fairly normal. But cmd.exe is actually a
regression from MS-DOS in that it apparently will not allow this. Or it
is a hugh pain.

You could get a program that emulates a full-screen ANSI terminal, and
learn to use ANSI control codes. Or you could use a tkinter (tk) Text
widget. People have written at least serial terminal emulators for Text,
but I did not find a full-screen.

Using tkinter, I would try making each box a separate text box placed in
a frameand let tkinter worry about displaying them correctly and
detecting which box get a mouse click or <enter> key.

[Glenn Linderman]

Post by Terry Reedy

Post by Wiktor
Hi,
as OO programming exercise, I'm trying to port to Python one of my favorite
game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
version on C64 http://youtu.be/UFycYOp2cbE and here's

This appears to be an actual text screen, no graphics.

Post by Wiktor
video from modern rewritten (for Atari emulators) version: Kolony 2106
http://youtu.be/eX20Qqqm5eg - you get the idea? ;-)).

This appears to be text boxes on a graphics screen.

Post by Wiktor
OO Design is one thing, but I want to make it look as near as
possible to
the original (those windows-like menus in console window).

Which original? the C64 or Atari. The important characteristic of
both is that both have multiple overlapping popup boxes. This means
that either you or a widget framework much keep track of stacking
order and how to restore what was hidden when a box goes away or is
moved down in the stacking order. I would not be surprised if the
Atari had at least a rudimentary widget framework.

Post by Wiktor
I tried to use
'standard' Unicode characters (I can see that most of my Windows monospaced
????????????????
? Construction ?
? Production ?
? Research ?
? Exploration ?
????????????????
? Next turn ?
????????????????
(I like the look of double lines on right and at the bottom)
| File "E:\Moje dokumenty\python\kolony\menu.py", line 14, in <module>
| """
| File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
| return codecs.charmap_encode(input,self.errors,encoding_map)[0]
| UnicodeEncodeError: 'charmap' codec can't encode character '\u2556' in position 1
| 6: character maps to <undefined>

You have two separate problems with running in the windows console.
1. The character issue. If you run a program to just print the above
from an Idle editor, so that the output is printed to the Idle shell,
there should be no problem.

Post by Wiktor

print('\u2556'*10)

??????????
But characters are not your real issue.
2. The random access issue. The MS console in normal use is like a
serial printer or terminal. Once a line is printed, it cannot be
changed. I looked at the video and the program randomly accesses a 24
or 25 line x 80 column screen, overprinting existing characters at
will and reversing black on white versus white of black at will.
MSDOS screens recognized standard ANSI screen control codes once the
ANSI.SYS driver was installed, which was fairly normal. But cmd.exe is
actually a regression from MS-DOS in that it apparently will not allow
this. Or it is a hugh pain.
You could get a program that emulates a full-screen ANSI terminal, and
learn to use ANSI control codes. Or you could use a tkinter (tk) Text
widget. People have written at least serial terminal emulators for
Text, but I did not find a full-screen.
Using tkinter, I would try making each box a separate text box placed
in a frameand let tkinter worry about displaying them correctly and
detecting which box get a mouse click or <enter> key.

I've never used the API from Python but random console access is
documented at
http://msdn.microsoft.com/en-us/library/windows/desktop/ms687404%28v=vs.85%29.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140804/ac11075b/attachment.html>

[Wolfgang Maier]

Post by Glenn Linderman
I've never used the API from Python but random console access is
documented at
http://msdn.microsoft.com/en-us/library/windows/desktop/ms687404%28v=vs.85%29.aspx

Would using the API from Python involve doing the wrapping yourself or
do you know about an existing package for the job ?

By the way (and off-topic), how would you do it on Linux?

Wolfgang

[Glenn Linderman]

Post by Wolfgang Maier

Post by Glenn Linderman
I've never used the API from Python but random console access is
documented at
http://msdn.microsoft.com/en-us/library/windows/desktop/ms687404%28v=vs.85%29.aspx

Would using the API from Python involve doing the wrapping yourself or
do you know about an existing package for the job ?

I haven't used the API from Python. I haven't checked PyWin32 to see if
it already wraps that API like it wraps so many other APIs. I haven't
Googled using "python" and "WriteConsoleOutput" to see if other packages
may exist to do the job. But these are the things that I would do if I
had a need to write a program like yours, since I know that the console
does, in fact, support random access.

Post by Wolfgang Maier
By the way (and off-topic), how would you do it on Linux?

Off topic? It is still about doing it using Python, no?

I believe that most Unix terminal emulators, which are used for running
shells and command lines, support cursor controls, and I believe most of
them have a mode that emulates the DEC VT-52 terminal, one of which I
had physical access to at the "computer lab" at the university I
attended so many years ago. The VT-52 defined escape sequences to move
the cursor around on the screen, providing random access. Text-based,
screen-oriented programs such as emacs leveraged such capabilities quite
effectively.

There may be something better today, I haven't used Unix for a dozen
years now, and the usage at that time was database development not
text-based graphics. I've used Linux only on my web host, and a little
experimentation on a local machine I installed it on here, until the
machine died, and I didn't do any text-based graphics in either of those
circumstances either. So probably college was the last time I used
text-based graphics, but that was using RSTS and DECsystem 20 (forget
the name of the OS for that machine) on VT-52 terminals. But I've noted
with amusement that the VT-52 (and later enhanced models) are still
supported by Unix/Linux terminal emulators and X system.

Unix abstracts that cursor motion using "curses" and I believe there are
curses implementations for Windows as well, but I've not attempted to
use curses from Python on either Unix or Windows.

Post by Wolfgang Maier
Wolfgang

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140804/254ff746/attachment.html>

[Terry Reedy]

Post by Wolfgang Maier

Post by Glenn Linderman
I've never used the API from Python but random console access is
documented at
http://msdn.microsoft.com/en-us/library/windows/desktop/ms687404%28v=vs.85%29.aspx

Would using the API from Python involve doing the wrapping yourself or
do you know about an existing package for the job ?

The packages Wiktor is using, 'termcolor', 'colorama'
and 'colorconsole' - (all on PyPI), must be using WriteConsoleOutput.

[Grant Edwards]

Post by Glenn Linderman
I believe that most Unix terminal emulators, which are used for running
shells and command lines, support cursor controls, and I believe most of
them have a mode that emulates the DEC VT-52 terminal,

I'm not aware of any that are in common use, but there may be some
niche VT52 emulators somewhere I don't know about. All the widely
used terminal emulators are some flavor of ANSI (for you DEC guys,
VT100 and later) rather than VT52.

[Wiktor]

Hi,
first, thank you all for responses. I decided to just use single line frame
around menu. Yes, those double+single line corners work fine in ConEmu, but
I don't want this Python script to be dependent on external program. Maybe
one day it will be worth of showing to others, and it's silly to tell them
'It is pure console based game/framework but works only in ConEmu'...

Post by Terry Reedy

Post by Wiktor
as OO programming exercise, I'm trying to port to Python one of my favorite
game from early'90 (Atari 65XL/XE) - Kolony (here's video from original
version on C64 http://youtu.be/UFycYOp2cbE and here's

This appears to be an actual text screen, no graphics.

Post by Wiktor
video from modern rewritten (for Atari emulators) version: Kolony 2106
http://youtu.be/eX20Qqqm5eg - you get the idea? ;-)).

This appears to be text boxes on a graphics screen.

Post by Wiktor
OO Design is one thing, but I want to make it look as near as possible to
the original (those windows-like menus in console window).

Which original? the C64 or Atari. The important characteristic of both
is that both have multiple overlapping popup boxes. This means that
either you or a widget framework much keep track of stacking order and
how to restore what was hidden when a box goes away or is moved down in
the stacking order. I would not be surprised if the Atari had at least a
rudimentary widget framework.

Yes, I'm aware that first link is to the text based game, and second to
graphic based game. I provided both links, because I couldn't find screen
cast from original Atari game (which is also text based, but IMO looks
better than C64's version), and this modern game is translated to English,
so is better for you to understand character of game.
Yes, I'd like to make text game, that looks like window-based, with popup
boxes, inactive windows grayed out and all this stuff. And all this running
on standard console window (cmd.exe).

I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
and 'colorconsole' - they provide functions to print text at desired
position on screen, and change color of foreground/background of this text.
With those packages I already developed some classes that allow me to
construct some simple menus for my console programs. Short demo of silly
program calculating degree of n-th root:

(I link to the video, because there's too much code lines to paste them
here. Also it's dependent upon those third party packages, and still
work-in-progress).

So, I'm not worry about randomly access, colors, overprinting existing
characters. At this point I know how to do that.
I'm taking next step, so I tried to draw nice frame around menu (that's
why I posted yesterday).
Next step would be to manage those widgets to draw one over another, to
keep track which one window opens which, and when the other window must be
closed and when only grayed out. At this point I still don't know how to do
this right, but I'm thinking about this very hard. :-) Probably one day
I'll ask it here, if I don't figure it out. :-)

Wiktor

[Wolfgang Maier]

Post by Wiktor
Hi,
first, thank you all for responses. I decided to just use single line frame
around menu. Yes, those double+single line corners work fine in ConEmu, but
I don't want this Python script to be dependent on external program. Maybe
one day it will be worth of showing to others, and it's silly to tell them
'It is pure console based game/framework but works only in ConEmu'...
I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
and 'colorconsole' - they provide functions to print text at desired
position on screen, and change color of foreground/background of this text.

Thanks for pointing out these packages! Since you say you're using all
three of them: where do colorama and colorconsole differ. From a quick
look, I can see that termcolor is a bit different, but colorama and
colorconsole seem pretty similar in scope.

Thanks,
Wolfgang

[Wiktor]

Post by Wolfgang Maier

Post by Wiktor
I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
and 'colorconsole' - they provide functions to print text at desired
position on screen, and change color of foreground/background of this text.

Thanks for pointing out these packages! Since you say you're using all
three of them: where do colorama and colorconsole differ. From a quick
look, I can see that termcolor is a bit different, but colorama and
colorconsole seem pretty similar in scope.

From colorama I just use one function - init(). Without this
initialization all those ansii escape characters (used by colorama itself,
but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
make it work.
All coloring work I make in termcolor.colored() function, because it
returns string, which I can work on (store and/or send it to print_at()
function later).
And colorconsole is helpful with its screen.print_at() function [where
screen = colorconsole.terminal.get_terminal()].

So, yes, it's matter of (probably bad) design, but now I need all three
packages. Maybe if I resign from storing my colored strings, and color them
just while sending them to printing function, I could get rid of colorama
and termcolor...
Well, thanks for asking, because now, during writing this response, I see
that maybe redesign is worth of trying...

Wiktor

[Chris Angelico]

Post by Wiktor
From colorama I just use one function - init(). Without this
initialization all those ansii escape characters (used by colorama itself,
but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
make it work.

I dug into colorama's source code, and it seems that "just one
function" is a little dismissive :) When you call colorama's init(),
it replaces stdout with a wrapper that parses ANSI sequences and turns
them into API calls. So, yeah, without that anything that outputs ANSI
sequences isn't going to work.

ChrisA

[Wiktor]

Post by Chris Angelico

Post by Wiktor
From colorama I just use one function - init(). Without this
initialization all those ansii escape characters (used by colorama itself,
but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
make it work.

I dug into colorama's source code, and it seems that "just one
function" is a little dismissive :) When you call colorama's init(),
it replaces stdout with a wrapper that parses ANSI sequences and turns
them into API calls. So, yeah, without that anything that outputs ANSI
sequences isn't going to work.

Maybe I didn't write it clear. :-) What I meant was, that even though I
don't use any other functions from colorama (I color all the strings with
termcolor) - I still have to use init() function from colorama.
termcolor doesn't want to work alone, even though its described as OS
independent. I guess it works fine on Linux terminal without init()
function from colorama. In cmd.exe I need colorama just for this.

[Terry Reedy]

Post by Wiktor

Post by Chris Angelico

Post by Wiktor
From colorama I just use one function - init(). Without this
initialization all those ansii escape characters (used by colorama itself,
but also by termcolor.colored()) don't work in cmd.exe. At least I couldn't
make it work.

I dug into colorama's source code, and it seems that "just one
function" is a little dismissive :) When you call colorama's init(),
it replaces stdout with a wrapper that parses ANSI sequences and turns
them into API calls. So, yeah, without that anything that outputs ANSI
sequences isn't going to work.

Maybe I didn't write it clear. :-) What I meant was, that even though I
don't use any other functions from colorama (I color all the strings with
termcolor) - I still have to use init() function from colorama.
termcolor doesn't want to work alone, even though its described as OS
independent.

Termcolor says "ANSI Color formatting for output in terminal."
https://pypi.python.org/pypi/termcolor/1.1.0

It is OS-independent but depends on support of standard ANSI screen
command codes. Microsoft removed that support from cmd.exe. If you look
at the Terminal properties box on the page above, the only thing
termcolor can do on Windows, by itself, is reversed text. Colorama.init
adds back (at least some of) the ANSI to API translation omitted from
cmd.exe.

Post by Wiktor
I guess it works fine on Linux terminal without init()

Because linux terminals translate ANSI to whatever api calls are needed.

Post by Wiktor
function from colorama. In cmd.exe I need colorama just for this.

[Terry Reedy]

Post by Wiktor
Yes, I'd like to make text game, that looks like window-based, with popup
boxes, inactive windows grayed out and all this stuff. And all this running
on standard console window (cmd.exe).

Your problem doing this is that cmd.exe is not a standard since 30 years
ago full-screen console window, , but is intentionally crippled to stop
people from doing what you are trying to do. Some as MS would like to
delete it altogether.

Post by Wiktor
I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
and 'colorconsole' -

All on pypi.python.org, I see.

Post by Wiktor
they provide functions to print text at desired
position on screen, and change color of foreground/background of this text.
With those packages I already developed some classes that allow me to
construct some simple menus for my console programs. Short demo of silly
http://youtu.be/V8ilLhHAT_k

I am impressed with what the authors of those packages managed to do.

Post by Wiktor
I'm taking next step, so I tried to draw nice frame around menu (that's
why I posted yesterday).

Is there no working codepage with ascii text and the line chars? I
suppose I am not surprised if not.

Post by Wiktor
Next step would be to manage those widgets to draw one over another, to
keep track which one window opens which, and when the other window must be
closed and when only grayed out. At this point I still don't know how to do
this right, but I'm thinking about this very hard. :-) Probably one day
I'll ask it here, if I don't figure it out. :-)

You may have to settle for using different background colors to
delineate different boxes.

[Chris Angelico]

Is there no working codepage with ascii text and the line chars? I suppose I
am not surprised if not.

That would be codepage 437. I grew up with that on DOS, as the one and
only 256-character set, and then when we moved to OS/2 and actual
codepages, we always configured 437,850 (that is, 437 as primary, with
850 as a secondary if we wanted it). Trouble is, it doesn't have
non-basic letters, and the OP needs to write Polish text. I'd be not
at all surprised if there are characters he needs that aren't in
CP437.

ChrisA

[Wiktor]

Post by Terry Reedy

Post by Wiktor
I'm taking next step, so I tried to draw nice frame around menu (that's
why I posted yesterday).

Is there no working codepage with ascii text and the line chars? I
suppose I am not surprised if not.

With single line (???????????) and double line (???????????) - many
codepages, CP852 for sure.

With corners/crosses where single and double lines meet (?????????
??????????) - I know only one: CP437.

But I can't have both - Polish letters and all those line chars, so I
can't do this fancy frame from first post with Polish strings inside. There
will be simpler version instead.

[Akira Li]

Post by Wiktor

Post by Terry Reedy

Post by Wiktor
I'm taking next step, so I tried to draw nice frame around menu (that's
why I posted yesterday).

Is there no working codepage with ascii text and the line chars? I
suppose I am not surprised if not.

With single line (???????????) and double line (???????????) - many
codepages, CP852 for sure.
With corners/crosses where single and double lines meet (?????????
??????????) - I know only one: CP437.
But I can't have both - Polish letters and all those line chars, so I
can't do this fancy frame from first post with Polish strings inside. There
will be simpler version instead.

Unicode has line drawing characters [1]. win_unicode_console [2] allows
to print Unicode in cmd.exe. win_unicode_console and colorama will
probably conflict. You could look at the source to see how hard to
combine both functionalities.

[1] http://en.wikipedia.org/wiki/Box-drawing_character
[2] https://pypi.python.org/pypi/win_unicode_console

btw, blessings [3] provides an easy-to-use interface if you need to add
colors and move text in a terminal. It claims that it also supports
colors on Windows if used with colorama.

[3] https://pypi.python.org/pypi/blessings/


--
Akira

[Wiktor]

Post by Akira Li
Unicode has line drawing characters [1]. win_unicode_console [2] allows
to print Unicode in cmd.exe. win_unicode_console and colorama will
probably conflict. You could look at the source to see how hard to
combine both functionalities.
[1] http://en.wikipedia.org/wiki/Box-drawing_character
[2] https://pypi.python.org/pypi/win_unicode_console
btw, blessings [3] provides an easy-to-use interface if you need to add
colors and move text in a terminal. It claims that it also supports
colors on Windows if used with colorama.
[3] https://pypi.python.org/pypi/blessings/

[2] - indeed does not work with colorconsole/colorama. And I'm not that
smart to combine those two functionalities. :-)
[3] - maybe it works with colorama (colorama for coloring, and blessing
for positioning text), but now I don't even use colorama. I use
colorconsole to color AND position text, and I find it very handy. Don't
think that blessing+colorama would be more easy-to-use.

Thanks, I really appreciate every proposition to fix original problem you
all are giving me - I check them all.
But you need to understand, that I'm already OK with those cmd.exe
limitations, and really not trying to achieve look of frame from first
post. I'm OK with those single-only and double-only lines.

Now my game would look like this:
Loading Image... [*]
and I think it's very neat.

Wiktor

*) it's mocup (I don't have 'window manager' yet, so I have to show/
activate/deactivate/etc. every window by hand), but still

[giacomo boffi]

Post by Wiktor
I'm not starting from scratch. I'm using packages 'termcolor', 'colorama'
and 'colorconsole'

the 'urwid' package could be useful for similar projects but <cite>
requires Linux, OSX, Cygwin or other unix-like OS</cite> so I guess
it's of no use for you...

ciao
g

[Christian Gollwitzer]

Post by Chris Angelico

Post by Wiktor
I have to ask - is there a way to make that original concept work? I know,
that CP437 has symbols "?", "?" and "?", but does not have polish letters -
and I need to display them too.

Yeah, that's exactly the problem with codepages :)
The best way to do it is to use the Unicode codepage,

Agreed.

Post by Chris Angelico
but cmd.exe just
plain has issues.

It's not cmd.exe, it's the terminal that is shit. You can't even
interactively resize the width in the standard terminal.

Post by Chris Angelico
There are underlying Windows APIs for displaying
text that have problems with astral characters (I think that's what it
is), so ultimately, you're largely stuck.
One option would be to render the whole thing graphically, abandoning
cmd.exe altogether. That would be how a lot of telnet and SSH clients
will do the work. Get a proper Unicode-supporting toolkit (Tkinter has
issues with astral characters too, AIUI), and yes, you'll have to do a
lot of work yourself.

Tkinter only supports the BMP currently. But neither Polish nor box
drawing does require more: All those box drawing symbols are in the BMP:

http://www.unicode.org/charts/PDF/U2500.pdf

So you could use a Tkinter text widget and put your data there - or even
a simple label would do.

Christian

[Grant Edwards]

Post by Wiktor
okumenty\python\kolony\menu.py", line 14, in <module>

---8<-----------------
# coding:utf-8
test = """
????????????????
? Construction ?
? Production ?
? Research ?
? Exploration ?
????????????????
? Next turn ?
????????????????
"""

But can you do Polish as well? IIRC, that was the catch: encodings
that the windows terminal emulator understood that could do Polish
couldn't do double-lines and vice-versa.

I was a bit surprised when all that stuff rendered properly in slrn
runnnig in a terminal emulator. I must have done a better job with
locales and fonts that I thought...

[Wiktor]

Post by Wiktor
okumenty\python\kolony\menu.py", line 14, in <module>

I believe you, but I use Windows and its cmd.exe (as mentioned in
subject).

[wxjmfauth]

It works on *your* linux, because *your* linux is
automagically configured in the way the device that
hosts "unicode" (the console in that case) "works
in utf-8" (most probably).

On a "Russian" linux using koi8 or a "French" linux
using iso-8859-15, the problem is exacly similar as
on Windows. This is valid for any OS.

In essence, assuming the hosting device is "in utf-8"
is as wrong as assuming it is in, eg. cp852 or any
encoding.

This has nothing to do with Unicode.

jmf