Discussion:
Python Gotcha's?
(too old to reply)
Miki Tebeka
2012-04-04 22:34:20 UTC
Permalink
Greetings,

I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.

(Note that I want over http://wiki.python.org/moin/PythonWarts already).

Thanks,
--
Miki
Steven D'Aprano
2012-04-04 23:07:51 UTC
Permalink
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work. If you have an
interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
The GIL prevents Python from taking advantage of multiple cores in your
CPU when using multiple threads.

Solution: use a GIL-less Python, like IronPython or Jython, or use
multiple processes instead of threads.



exec() and execfile() are unintuitive if you supply separate dicts for
the globals and locals arguments.

http://bugs.python.org/issue1167300
http://bugs.python.org/issue14049

Note that both of these are flagged as WON'T FIX.

Solution: to emulate top-level code, pass the same dict as globals and
locals.


max() and min() fail with a single argument:
max(2, 3) => 3
max(3) => raises exception

Solution: don't do that. Or pass a list:
max([2, 3]) => 3
max([3]) => 3


Splitting on None and splitting on space is not identical:
"".split() => []
"".split(' ') => ['']



JSON expects double-quote marks, not single:
v = json.loads("{'test':'test'}") fails
v = json.loads('{"test":"test"}') succeeds




If you decorate a function, by default the docstring is lost.

@decorate
def spam(x, y):
"""blah blah blah blah"""

spam.__doc__ => raises exception

Solution: make sure your decorator uses functools.wraps().
--
Steven
Cameron Simpson
2012-04-04 23:21:21 UTC
Permalink
On 04Apr2012 23:07, Steven D'Aprano <steve+comp.lang.python at pearwood.info> wrote:
| If you decorate a function, by default the docstring is lost.
|
| @decorate
| def spam(x, y):
| """blah blah blah blah"""
|
| spam.__doc__ => raises exception
|
| Solution: make sure your decorator uses functools.wraps().

There's a functools.wraps()? [*smacks forehead with palm*] Thanks.
--
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

A pessimist is an optimist in full possession of the facts.
Duncan Booth
2012-04-05 11:06:11 UTC
Permalink
Post by Steven D'Aprano
v = json.loads("{'test':'test'}") fails
v = json.loads('{"test":"test"}') succeeds
You mean JSON expects a string with valid JSON?
Quelle surprise.
--
Duncan Booth http://kupuguy.blogspot.com
Chris Angelico
2012-04-05 11:37:04 UTC
Permalink
On Thu, Apr 5, 2012 at 9:06 PM, Duncan Booth
Post by Duncan Booth
? ? v = json.loads("{'test':'test'}") ?fails
? ? v = json.loads('{"test":"test"}') ?succeeds
You mean JSON expects a string with valid JSON?
Quelle surprise.
The surprise is between JSON and JavaScript (the latter does accept
single quotes), and Python is correctly implementing the JSON spec.

ChrisA
Steven D'Aprano
2012-04-05 12:00:47 UTC
Permalink
v = json.loads("{'test':'test'}") fails v =
json.loads('{"test":"test"}') succeeds
You mean JSON expects a string with valid JSON? Quelle surprise.
No. The surprise is that there exists a tool invented in the 21st century
that makes a distinction between strings quoted with " and those quoted
with '. Being used to a sensible language like Python, it boggled my
brain the first time I tried to write some JSON and naturally treated the
choice of quote mark as arbitrary. It especially boggled my brain when I
saw the pathetically useless error message generated:

py> json.loads("{'test':'test'}")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.2/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.2/json/decoder.py", line 351, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.2/json/decoder.py", line 367, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 1 (char 1)

"Expecting property name"??? WTF???


The reason this is a Gotcha rather than a bug is because the JSON
standard specifies the behaviour (probably in order to be compatible with
Javascript). Hence, although the behaviour is mind-numbingly stupid, it
is deliberate and not a bug. Hence, a gotcha.
--
Steven
Roy Smith
2012-04-05 12:32:10 UTC
Permalink
In article <4f7d896f$0$29983$c3e8da3$5496439d at news.astraweb.com>,
Post by Steven D'Aprano
You mean JSON expects a string with valid JSON? Quelle surprise.
No. The surprise is that there exists a tool invented in the 21st century
that makes a distinction between strings quoted with " and those quoted
with '. Being used to a sensible language like Python, it boggled my
brain the first time I tried to write some JSON and naturally treated the
choice of quote mark as arbitrary.
Your brain has a low boggle threshold.

There's absolutely no reason why JSON should follow Python syntax rules.
Making it support either kind of quotes would have complicated every
JSON library in the world, for no added value. Nobody should ever be
hand-writing JSON (just like nobody should ever be hand-writing XML).
Just use the supplied library calls and you'll never have to worry about
low-level minutia like this again.
Post by Steven D'Aprano
It especially boggled my brain when I
py> json.loads("{'test':'test'}")
[...]
ValueError: Expecting property name: line 1 column 1 (char 1)
"Expecting property name"??? WTF???
One of the hardest things about writing parsers is generating helpful
error messages when things don't parse. But, it's only of value to do
that when you're parsing something you expect to be written by a human,
and thus a human has to puzzle out what they did wrong. Nobody expects
that a JSON parser will be parsing human-written input, so there's
little value to saying anything more than "parse error".
Post by Steven D'Aprano
The reason this is a Gotcha rather than a bug is because the JSON
standard specifies the behaviour (probably in order to be compatible with
Javascript).
Well, considering that the JS in JSON stands for JavaScript...
Post by Steven D'Aprano
Hence, although the behaviour is mind-numbingly stupid, it
is deliberate and not a bug. Hence, a gotcha.
But surely not a Python gotcha. If anything, it's a JSON gotcha. The
same is true with PHP's JSON library, and Ruby's, and Perl's, and
Scala's, and C++'s, and so on. It's a JSON issue, and a silly one to be
complaining about at that.
Steve Howell
2012-04-05 14:32:19 UTC
Permalink
[...]?Nobody expects
that a JSON parser will be parsing human-written input, [...]
Humans write JSON all the time. People use JSON as a configuration
language, and some people actually write JSON files by hand. A common
example would be writing package.json for an npm package.

Here are a couple examples:

https://github.com/jashkenas/coffee-script/blob/master/package.json
https://github.com/github/hubot/blob/master/package.json
so there's little value to saying anything more than "parse error".
So, there's little value to say anything more than "parse
error"...except to help all those dumb humans that expect JSON to be
human-writable. ;)
Grzegorz Staniak
2012-04-05 16:25:16 UTC
Permalink
Post by Roy Smith
There's absolutely no reason why JSON should follow Python syntax
rules. Making it support either kind of quotes would have
complicated every JSON library in the world, for no added value.
I think these days it's not just "Python syntax", it's kinda something
that you can get accustommed to take for granted. Realistically, how
much more complication could the support for either quote marks
introduce? I doubt anyone would even notice. And you don't have to
write JSON by hand for this gotcha to bite you, all it takes is to
start playing with generating JSON without the use of specialized
JSON libraries/functions. For testing, for fun, out of curiosity...

GS
--
Grzegorz Staniak <gstaniak _at_ gmail [dot] com>
Chris Angelico
2012-04-05 16:47:14 UTC
Permalink
Post by Grzegorz Staniak
Post by Roy Smith
There's absolutely no reason why JSON should follow Python syntax
rules. Making it support either kind of quotes would have
complicated every JSON library in the world, for no added value.
I think these days it's not just "Python syntax", it's kinda something
that you can get accustommed to take for granted. Realistically, how
much more complication could the support for either quote marks
introduce? I doubt anyone would even notice. And you don't have to
write JSON by hand for this gotcha to bite you, all it takes is to
start playing with generating JSON without the use of specialized
JSON libraries/functions. For testing, for fun, out of curiosity...
It all depends on the language. C and several C-derived languages
distinguish between "string constants" and 'integer constants', where
the latter are defined by character codepoint; PHP and bash have
"interpolated strings" and 'non-interpolated strings'; Python and REXX
have no difference between the two. All are legitimate design choices.

Assuming that multiple languages/protocols have the same flexibility
is dangerous, and I'm not surprised that JSON's strictness is biting
people. Maybe the weird error message should be addressed as a bug,
and then there won't be a problem :)

ChrisA
Roy Smith
2012-04-05 18:07:23 UTC
Permalink
In article <jlkh1c$msc$1 at mx1.internetia.pl>,
Post by Grzegorz Staniak
Post by Roy Smith
There's absolutely no reason why JSON should follow Python syntax
rules. Making it support either kind of quotes would have
complicated every JSON library in the world, for no added value.
I think these days it's not just "Python syntax", it's kinda something
that you can get accustommed to take for granted. Realistically, how
much more complication could the support for either quote marks
introduce? I doubt anyone would even notice. And you don't have to
write JSON by hand for this gotcha to bite you, all it takes is to
start playing with generating JSON without the use of specialized
JSON libraries/functions. For testing, for fun, out of curiosity...
If you want to talk a protocol, read the protocol specs and follow them.
Don't just look at a few examples, guess about the rules, and then act
surprised when your guess turns out to be wrong.

If you don't want to take the trouble to read and understand the
protocol specs, use a library written by somebody who has already done
the hard work for you.
Grzegorz Staniak
2012-04-05 20:18:22 UTC
Permalink
Post by Roy Smith
Post by Grzegorz Staniak
Post by Roy Smith
There's absolutely no reason why JSON should follow Python syntax
rules. Making it support either kind of quotes would have
complicated every JSON library in the world, for no added value.
I think these days it's not just "Python syntax", it's kinda something
that you can get accustommed to take for granted. Realistically, how
much more complication could the support for either quote marks
introduce? I doubt anyone would even notice. And you don't have to
write JSON by hand for this gotcha to bite you, all it takes is to
start playing with generating JSON without the use of specialized
JSON libraries/functions. For testing, for fun, out of curiosity...
If you want to talk a protocol, read the protocol specs and follow them.
Sure, sure. But it still may raise a few eyebrows as people start to
play along while still reading the spces. It's just not something that
I'd expect (yes, I learnt Perl before I discovered Python).

GS
--
Grzegorz Staniak <gstaniak _at_ gmail [dot] com>
Tycho Andersen
2012-04-05 16:31:56 UTC
Permalink
Post by Roy Smith
One of the hardest things about writing parsers is generating helpful
error messages when things don't parse. But, it's only of value to do
that when you're parsing something you expect to be written by a human,
and thus a human has to puzzle out what they did wrong. Nobody expects
that a JSON parser will be parsing human-written input, so there's
little value to saying anything more than "parse error".
Except for the human that has to debug why something automatically
generated doesn't parse. That guy would probably appreciate a
reasonable error message. (And indeed, as a sibling poster points out,
people do write JSON by hand quite frequently.)

\t
Steven D'Aprano
2012-04-05 18:15:46 UTC
Permalink
Post by Roy Smith
In article <4f7d896f$0$29983$c3e8da3$5496439d at news.astraweb.com>,
Post by Steven D'Aprano
You mean JSON expects a string with valid JSON? Quelle surprise.
No. The surprise is that there exists a tool invented in the 21st
century that makes a distinction between strings quoted with " and
those quoted with '. Being used to a sensible language like Python, it
boggled my brain the first time I tried to write some JSON and
naturally treated the choice of quote mark as arbitrary.
Your brain has a low boggle threshold.
There's absolutely no reason why JSON should follow Python syntax rules.
Except for the most important reason of all: Python's use of alternate
string delimiters is an excellent design, one which Javascript itself
follows.

http://www.javascripter.net/faq/quotesin.htm

I'm not the only one who has had trouble with JSON's poor design choice:

http://stackoverflow.com/a/4612914

For a 21st century programming language or data format to accept only one
type of quotation mark as string delimiter is rather like having a 21st
century automobile with a hand crank to start the engine instead of an
ignition. Even if there's a good reason for it (which I doubt), it's
still surprising.
Post by Roy Smith
Making it support either kind of quotes would have complicated every
JSON library in the world, for no added value.
Ooooh, such complication. I wish my software was that complicated.

The added value includes:

* semantic simplicity -- a string is a string, regardless of which
quotes are used for delimiters;

* reducing the number of escaped quotes needed;

* compatibility with Javascript;

* robustness.

As it stands, JSON fails to live up to the Robustness principle and
Postel's law:

Be liberal in what you accept, and conservative in what you send.


http://en.wikipedia.org/wiki/Robustness_principle
Post by Roy Smith
Nobody should ever be hand-writing JSON
So you say, but it is a fact that people do. And even if they don't hand-
write it, people do *read* it, and allowing both quotation marks aids
readability:

"\"Help me Obiwan,\" she said, \"You're my only hope!\""

Blah. You can cut the number of escapes needed to one:

'"Help me Obiwan," she said, "You\'re my only hope!"'
Post by Roy Smith
(just like nobody should ever be hand-writing XML).
Just use the supplied library calls and you'll never have to worry about
low-level minutia like this again.
Post by Steven D'Aprano
It especially boggled my brain when I saw the pathetically useless
py> json.loads("{'test':'test'}")
[...]
ValueError: Expecting property name: line 1 column 1 (char 1)
"Expecting property name"??? WTF???
One of the hardest things about writing parsers is generating helpful
error messages when things don't parse. But, it's only of value to do
that when you're parsing something you expect to be written by a human,
Or when debugging a faulty or buggy generator, or when dealing with non-
conforming or corrupted data. Essentially any time that you expect the
error message will be read by a human being. Which is always.

Error messages are for the human reader, always and without exception. If
you don't expect it to be read by a person, why bother with a message?
--
Steven
André Malo
2012-04-05 21:08:11 UTC
Permalink
Post by Steven D'Aprano
For a 21st century programming language or data format to accept only one
type of quotation mark as string delimiter is rather like having a 21st
century automobile with a hand crank to start the engine instead of an
ignition. Even if there's a good reason for it (which I doubt), it's
still surprising.
Here's a reason: KISS. Actually I've never understood the reason for
multiple equivalent quote characters. There are languages where these are
not equivalent, like perl, C or shell script. There it makes way more
sense.

(If a parser doesn't accept foreign syntax, that's reasonable enough for me,
too.)

nd
Cameron Simpson
2012-04-06 00:15:26 UTC
Permalink
On 05Apr2012 23:08, Andr? Malo <ndparker at gmail.com> wrote:
| * Steven D'Aprano wrote:
|
| > For a 21st century programming language or data format to accept only one
| > type of quotation mark as string delimiter is rather like having a 21st
| > century automobile with a hand crank to start the engine instead of an
| > ignition. Even if there's a good reason for it (which I doubt), it's
| > still surprising.
|
| Here's a reason: KISS. Actually I've never understood the reason for
| multiple equivalent quote characters. There are languages where these are
| not equivalent, like perl, C or shell script. There it makes way more
| sense.

I agree, except that even where the quotes are equivalent there is an
advantage: writing a string containing quotes:

'"Hello," he said.'
"It's funny weather."

Of course this doesn't scale much if you need both, but it covers a
common use.

Personally I'm ok with JSON accepting only one kind of quote.
And I _have_ been bitten by it and surprised in the past.
--
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

This telephone has too many shortcomings to be seriously considered as a
means of communication. The device is inherently of no value to us.
- Western Union memo, 1877
Steven D'Aprano
2012-04-06 01:03:45 UTC
Permalink
Post by André Malo
Post by Steven D'Aprano
For a 21st century programming language or data format to accept only
one type of quotation mark as string delimiter is rather like having a
21st century automobile with a hand crank to start the engine instead
of an ignition. Even if there's a good reason for it (which I doubt),
it's still surprising.
Here's a reason: KISS.
KISS is a reason *for* allowing multiple string delimiters, not against
it. The simplicity which matters here are:

* the user doesn't need to memorise which delimiter is allowed, and
which is forbidden, which will be different from probably 50% of
the other languages he knows;

* the user can avoid the plague of escaping quotes inside strings
whenever he needs to embed the delimiter inside a string literal.

This is the 21st century, not 1960, and if the language designer is
worried about the trivially small extra effort of parsing ' as well as "
then he's almost certainly putting his efforts in the wrong place.
--
Steven
Steve Howell
2012-04-06 01:20:49 UTC
Permalink
On Apr 5, 6:03?pm, Steven D'Aprano <steve
Post by Steven D'Aprano
Post by André Malo
Post by Steven D'Aprano
For a 21st century programming language or data format to accept only
one type of quotation mark as string delimiter is rather like having a
21st century automobile with a hand crank to start the engine instead
of an ignition. Even if there's a good reason for it (which I doubt),
it's still surprising.
Here's a reason: KISS.
KISS is a reason *for* allowing multiple string delimiters, not against
* the user doesn't need to memorise which delimiter is allowed, and
? which is forbidden, which will be different from probably 50% of
? the other languages he knows;
Exactly. One of the reasons that human use computers in the first
place is that we have flawed memory with respect to details,
especially arbitrary ones. It's the job of the computer to make our
lives easier.
Post by Steven D'Aprano
* the user can avoid the plague of escaping quotes inside strings
? whenever he needs to embed the delimiter inside a string literal.
Unlike JSON, JS itself allows '"' and "'", although its canonical
representation of the latter is '\''.

Having said that, I don't mind that JSON is strict; I just hate that
certain JSON parsers give cryptic messages on such an obvious gotcha.
Post by Steven D'Aprano
This is the 21st century, not 1960, and if the language designer is
worried about the trivially small extra effort of parsing ' as well as "
then he's almost certainly putting his efforts in the wrong place.
FWIW the JSON parser in Javascript is at least capable of giving a
precise explanation in its error message, which put it ahead of
Post by Steven D'Aprano
config = "{'foo': 'bar'}"
"{'foo': 'bar'}"
Post by Steven D'Aprano
JSON.parse(config)
SyntaxError: Unexpected token '

(Tested under Chrome and node.js, both based on V8.)
Post by Steven D'Aprano
Post by André Malo
Post by Steven D'Aprano
config = "{'foo': 'bar'}"
import json
json.loads(config)
[...]
ValueError: Expecting property name: line 1 column 1 (char 1)

(Python's implementation at least beats JS by including line/column
info.)
Dan Sommers
2012-04-06 01:54:30 UTC
Permalink
On 06 Apr 2012 01:03:45 GMT
This is the 21st century, not 1960 ...
Now there's a slippery slope, indeed. ;-)
... and if the language designer is worried about the trivially small
extra effort of parsing ' as well as " then he's almost certainly
putting his efforts in the wrong place.
What about ? and ?, or ? and ?, or ? and ??

Any unicode-aware parser should be able to determine that the next input
character is "punctuation, open" or "punctuation, initial" (and doesn't
already mean something else in the language) and then scan for the a
corresponding "punctuation, close" or "punctuation, final" character.

Dan
André Malo
2012-04-06 15:40:41 UTC
Permalink
Post by Steven D'Aprano
Post by André Malo
Post by Steven D'Aprano
For a 21st century programming language or data format to accept only
one type of quotation mark as string delimiter is rather like having a
21st century automobile with a hand crank to start the engine instead
of an ignition. Even if there's a good reason for it (which I doubt),
it's still surprising.
Here's a reason: KISS.
KISS is a reason *for* allowing multiple string delimiters, not against
* the user doesn't need to memorise which delimiter is allowed, and
which is forbidden, which will be different from probably 50% of
the other languages he knows;
* the user can avoid the plague of escaping quotes inside strings
whenever he needs to embed the delimiter inside a string literal.
This is the 21st century, not 1960, and if the language designer is
worried about the trivially small extra effort of parsing ' as well as "
then he's almost certainly putting his efforts in the wrong place.
Yes, that's what you said already. My reasoning was in the part you stripped
from my quote. *shrug*

nd
--
Treat your password like your toothbrush. Don't let anybody else
use it, and get a new one every six months. -- Clifford Stoll

(found in ssl_engine_pphrase.c)
rusi
2012-04-06 15:43:55 UTC
Permalink
Post by André Malo
Post by Steven D'Aprano
Post by André Malo
Post by Steven D'Aprano
For a 21st century programming language or data format to accept only
one type of quotation mark as string delimiter is rather like having a
21st century automobile with a hand crank to start the engine instead
of an ignition. Even if there's a good reason for it (which I doubt),
it's still surprising.
Here's a reason: KISS.
KISS is a reason *for* allowing multiple string delimiters, not against
* the user doesn't need to memorise which delimiter is allowed, and
? which is forbidden, which will be different from probably 50% of
? the other languages he knows;
* the user can avoid the plague of escaping quotes inside strings
? whenever he needs to embed the delimiter inside a string literal.
This is the 21st century, not 1960, and if the language designer is
worried about the trivially small extra effort of parsing ' as well as "
then he's almost certainly putting his efforts in the wrong place.
Yes, that's what you said already. My reasoning was in the part you stripped
from my quote. *shrug*
Multiple symmetric quote characters breaks one of python's own zen
rules:

There should be one-- and preferably only one --obvious way to do it.
Grzegorz Staniak
2012-04-06 16:02:49 UTC
Permalink
Post by rusi
Post by André Malo
Post by Steven D'Aprano
This is the 21st century, not 1960, and if the language designer is
worried about the trivially small extra effort of parsing ' as well as "
then he's almost certainly putting his efforts in the wrong place.
Yes, that's what you said already. My reasoning was in the part you stripped
from my quote. *shrug*
Multiple symmetric quote characters breaks one of python's own zen
There should be one-- and preferably only one --obvious way to do it.
Then again, practicality beats purity.

GS
--
Grzegorz Staniak <gstaniak _at_ gmail [dot] com>
André Malo
2012-04-08 20:43:46 UTC
Permalink
Post by Grzegorz Staniak
Post by rusi
There should be one-- and preferably only one --obvious way to do it.
Then again, practicality beats purity.
Yes.

If you ever grepped for, say, the usage of dictionary keys in a bigger
application, you might agree, that having multiple equivalent quote
characters is not as practical as it might seem in the first place.

Code is written once and read often.

nd
Roy Smith
2012-04-05 21:20:11 UTC
Permalink
In article <4f7de152$0$29983$c3e8da3$5496439d at news.astraweb.com>,
This is getting a bit off-topic. If you wish to argue that JSON is
designed poorly, you should do that in some appropriate JSON forum.
It's not a Python issue.

Now, if you wish to boggle your mind about something pythonic, how about
mutexes not being thread safe (http://bugs.python.org/issue1746071)?
Miki Tebeka
2012-04-05 23:36:35 UTC
Permalink
Post by Roy Smith
Now, if you wish to boggle your mind about something pythonic, how about
mutexes not being thread safe (http://bugs.python.org/issue1746071)?
This is and old and deprecated module, you should not use it.

Use http://docs.python.org/library/threading.html#threading.Lock and friends instead.

IMO the bug should be to remove this module and it's documentation :)
Terry Reedy
2012-04-06 01:46:57 UTC
Permalink
Post by Miki Tebeka
Post by Roy Smith
Now, if you wish to boggle your mind about something pythonic, how about
mutexes not being thread safe (http://bugs.python.org/issue1746071)?
This is and old and deprecated module, you should not use it.
Use http://docs.python.org/library/threading.html#threading.Lock and friends instead.
IMO the bug should be to remove this module and it's documentation :)
Post by Roy Smith
import mutex
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import mutex
ImportError: No module named 'mutex'

It was closed as "won't fix" because
"On 2008/2/23, Guido van Rossum said in python-dev
Post by Miki Tebeka
According to the docstring it's only meant to be used with sched.py.
Please don't try to make it work with threads!
Anyway, this module will be removed, or at least its API hidden, in 3.0."
--
Terry Jan Reedy
Tim Roberts
2012-04-08 03:47:15 UTC
Permalink
Post by Roy Smith
There's absolutely no reason why JSON should follow Python syntax rules.
No, but there certainly is a justification for expecting JAVASCRIPT Object
Notation (which is, after all, what JSON stands for) to follow Javascript's
syntax rules. And Javascript happens to follow the same quoting rules as
Python.

Now, I fully understand that it is the way it is. I'm merely pointing out
that his was not an unreasonable expectation.
--
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.
Chris Angelico
2012-04-08 03:54:23 UTC
Permalink
Post by Tim Roberts
No, but there certainly is a justification for expecting JAVASCRIPT Object
Notation (which is, after all, what JSON stands for) to follow Javascript's
syntax rules. ?And Javascript happens to follow the same quoting rules as
Python.
Now, I fully understand that it is the way it is. ?I'm merely pointing out
that his was not an unreasonable expectation.
I agree, it would make sense. But that's only because of the name; if
it had been called "Just Simple Object Notation" or something
stupider, then nobody would expect it to be the same as anything. And
these days, nobody's confused by the fact that Java and Javascript are
completely different beasts.

ChrisA
Tim Wintle
2012-04-05 12:41:05 UTC
Permalink
Post by Steven D'Aprano
The reason this is a Gotcha rather than a bug is because the JSON
standard specifies the behaviour (probably in order to be compatible with
Javascript).
It's not to be compatible with javascript (you can use either in
javascript)

I believe the choice is to make the parser as simple as possible. Agreed
it's a gotcha, but json is almost always generated automatically.

Tim
Steve Howell
2012-04-05 14:13:22 UTC
Permalink
On Apr 5, 5:00?am, Steven D'Aprano <steve
Post by Steven D'Aprano
? ? v = json.loads("{'test':'test'}") ?fails v =
? ? json.loads('{"test":"test"}') ?succeeds
You mean JSON expects a string with valid JSON? Quelle surprise.
No. The surprise is that there exists a tool invented in the 21st century
that makes a distinction between strings quoted with " and ?those quoted
with '. Being used to a sensible language like Python, it boggled my
brain the first time I tried to write some JSON and naturally treated the
choice of quote mark as arbitrary.
I've been bitten by this gotcha too. Maybe "boggled my brain" would
be a bit of hyperbole, but it did cause me minor pain, and brief but
frustrating pain is the whole point of "gotcha" presentations.
Post by Steven D'Aprano
It especially boggled my brain when I
[...]
ValueError: Expecting property name: line 1 column 1 (char 1)
"Expecting property name"??? WTF???
I agree with you that the error message is pretty puzzling. I can
understand the rationale of the parser authors not to go overboard
with diagnosing these errors correctly to users, since it would
complicate the parser code and possibly slow it down even for well
formed JSON. On the other hand, I think that parsers can distinguish
themselves by anticipating the most common gotchas and giving clear
messages.
Post by Steven D'Aprano
The reason this is a Gotcha rather than a bug is because the JSON
standard specifies the behaviour (probably in order to be compatible with
Javascript). Hence, although the behaviour is mind-numbingly stupid, it
is deliberate and not a bug. Hence, a gotcha.
Yep.
Steven D'Aprano
2012-04-05 18:22:22 UTC
Permalink
v = json.loads("{'test':'test'}") fails v =
json.loads('{"test":"test"}') succeeds
You mean JSON expects a string with valid JSON? Quelle surprise.
Actually, on further thought, and on reading the JSON RFC, I have decided
that this is a design bug and not merely a gotcha.

The relevant section of the RFC is this:


4. Parsers

A JSON parser transforms a JSON text into another representation. A
JSON parser MUST accept all texts that conform to the JSON grammar.
A JSON parser MAY accept non-JSON forms or extensions.


http://www.ietf.org/rfc/rfc4627.txt


So a valid parser is permitted to accept data which is not strictly JSON.
Given that both Javascript and Python (and I would argue, any sensible
modern language) allows both single and double quotation marks as
delimiters, the JSON parser should do the same. Failure to take advantage
of that is a design flaw.

Of course, the RFC goes on to say that a JSON generator MUST only
generate text which conforms to the JSON grammar. So a conforming
implementation would be perfectly entitled to accept, but not emit,
single-quote delimited strings.
--
Steven
rusi
2012-04-06 04:28:01 UTC
Permalink
Post by Duncan Booth
? ? v = json.loads("{'test':'test'}") ?fails
? ? v = json.loads('{"test":"test"}') ?succeeds
You mean JSON expects a string with valid JSON?
Quelle surprise.
Are there languages (other than python) in which single and double
quotes are equivalent?

[No I dont claim to know all the languages out there, just that I dont
know any other language which allows single and double quotes to be
interconvertible like python does]
Chris Angelico
2012-04-06 04:35:21 UTC
Permalink
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
[No I dont claim to know all the languages out there, just that I dont
know any other language which allows single and double quotes to be
interconvertible like python does]
REXX and JavaScript. And probably others.

ChrisA
Steve Howell
2012-04-06 04:54:06 UTC
Permalink
Post by rusi
Post by Duncan Booth
? ? v = json.loads("{'test':'test'}") ?fails
? ? v = json.loads('{"test":"test"}') ?succeeds
You mean JSON expects a string with valid JSON?
Quelle surprise.
Are there languages (other than python) in which single and double
quotes are equivalent?
[No I dont claim to know all the languages out there, just that I dont
know any other language which allows single and double quotes to be
interconvertible like python does]
Python doesn't treat single quotes and double quotes in *exactly* the
same manner, because the choice of outer quotes affects whether you
need to escape the outer quote characters inside the string. But I
don't want to be overly literal--I think I know what you mean by
"equivalent" here.

JS, YAML, and HTML are pretty similar to Python with respect to single
vs. double, as far as I know/remember/care.

Perl, Ruby, and CoffeeScript have the tradition that single quotes are
interpreted literally, whereas double quotes allow for interpolation
of things within the string. This is roughly inspired by English,
where one says things like "Double quotes are an 'enclosing syntax'
for single quotes." [Yes, I'm just making that up. Sounds plausible,
right?]

Both Ruby and CoffeeScript support triple quote syntax similar to
Python.

C uses double quotes for strings, as opposed to single quotes for
characters.

Java only allows double quotes for strings.

I'll wager a guess that if you took any two programming languages
(including declarative languages like SQL/HTML) and compared how they
represented string literals, there would be at least one thing
different between them, and that difference would be a fairly
arbitrary design decision. There are probably exceptions, but
languages that have the exact same quoting rules would probably be
close dialects of each other in other respects beyond quoting.

I'll also wager a guess that at least one thing I said above was
wrong, and that's a testament to the arcane nature of representing
string literals (as well as my own lack of mental capacity for
juggling all these different rules in my brain). And that's just in
ASCII with an American English bias. Throw in Unicode--that's when
things get really confusing!

I'm happy to stand corrected on any fact above. Withhold insults,
though. I already know that string literal syntax makes me feel
stupid--no need to rub it in.
rusi
2012-04-06 05:36:33 UTC
Permalink
Post by Steve Howell
JS, YAML, and HTML are pretty similar to Python with respect to single
vs. double, as far as I know/remember/care.
[Complete ignoramus here -- writing after a few minutes of googling]

YAML: http://yaml.org/spec/current.html#single%20quoted%20style/syntax
seems to indicate that the double-quote is multi-lineable the single
not.
[So YAML double-quote is like python triple-quote]

JS:
http://stackoverflow.com/questions/242813/when-to-use-double-or-single-quotes-in-javascript
seems to have some arcane argument about whether there is a difference
or not
Steve Howell
2012-04-06 05:50:30 UTC
Permalink
Post by rusi
Post by Steve Howell
JS, YAML, and HTML are pretty similar to Python with respect to single
vs. double, as far as I know/remember/care.
[Complete ignoramus here -- writing after a few minutes of googling]
YAML:http://yaml.org/spec/current.html#single%20quoted%20style/syntax
seems to indicate that the double-quote is multi-lineable the single
not.
[So YAML double-quote is like python triple-quote]
JS:http://stackoverflow.com/questions/242813/when-to-use-double-or-singl...
seems to have some arcane argument about whether there is a difference
or not
I like the fact that even this seemingly harmless statement on the
stackoverflow thread invites (light-hearted) controversy:

Double quotes will wear your shift key out faster :)

The response is this:

Not on Azerty, buddy.

And, yes, I am quoting via indentation here. But quoting is not a
"gotcha". Not in the least bit. Anybody who says "quoting is a
gotcha" just doesn't understand how simple this whole quoting business
is. Quoting is completely straightforward. "Just read the specs," I
think some wise person said earlier in this thread. They're
completely correct, of course. No gotchas here. Move along...
mwilson
2012-04-06 08:47:09 UTC
Permalink
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
Kernighan and Plauger's RATFOR (a pre-processor that added some C-like
syntax to FORTRAN) did that. Published in their book _Software Tools_.

Mel.
Roy Smith
2012-04-06 12:40:44 UTC
Permalink
Post by mwilson
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
Kernighan and Plauger's RATFOR (a pre-processor that added some C-like
syntax to FORTRAN) did that. Published in their book _Software Tools_.
I used to write a lot of code in RATFOR. It was really a pretty good
tool.
mwilson
2012-04-06 13:20:42 UTC
Permalink
Post by Roy Smith
Post by mwilson
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
Kernighan and Plauger's RATFOR (a pre-processor that added some C-like
syntax to FORTRAN) did that. Published in their book _Software Tools_.
I used to write a lot of code in RATFOR. It was really a pretty good
tool.
ISTR that RATFOR also concatenated adjacent quoted strings to build up long
strings. That sort of puts the capstone on the two-quote-characters scheme.
I can't lay my hands on the book to prove it.

GE/Honeywell FORTRAN stole the single quote to delimit seek addresses in I/O
statements, so my RATFOR implementations had to lose the two-quote feature
and use a two-for-one scheme, like backslash-escaping and % inclusion in
moduloed strings do in Python.

Mel.
Steven D'Aprano
2012-04-06 13:55:20 UTC
Permalink
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
Classic REXX, CSS, JavaScript, Lua, Prolog, XPath, YAML, Modula-2, HTML,
and (of course) English. There may be others.


Other languages like Perl, PHP and Ruby support alternate delimiters with
slightly different semantics.
--
Steven
Grzegorz Staniak
2012-04-06 14:18:19 UTC
Permalink
Post by Steven D'Aprano
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
Classic REXX, CSS, JavaScript, Lua, Prolog, XPath, YAML, Modula-2, HTML,
and (of course) English. There may be others.
Other languages like Perl, PHP and Ruby support alternate delimiters with
slightly different semantics.
Perl, first of all, has the 'q' and 'qq' operators. As much as I'd
come to dislike Perl after I discovered Python, I miss those two.
Every time I have to quote a string full of single/double quotes,
this comes to my mind:

q{'this' is not this, but 'that' is 'that' like 'this'}
q|'this' is not this, but 'that' is 'that' like 'this'|
q<'this' is not this, but 'that' is 'that' like 'this'>

... with 'qq' providing the version with inerpolation. I could
always find an arbitrary character for quoting that was _not_ present
in the string, and so, most of the time, avoid quoting altogether.
It was perhaps a bit too magical, but pruced very readable strings.

GS
--
Grzegorz Staniak <gstaniak _at_ gmail [dot] com>
rusi
2012-04-06 15:31:04 UTC
Permalink
Post by Grzegorz Staniak
Post by Steven D'Aprano
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
Classic REXX, CSS, JavaScript, Lua, Prolog, XPath, YAML, Modula-2, HTML,
and (of course) English. There may be others.
Other languages like Perl, PHP and Ruby support alternate delimiters with
slightly different semantics.
Perl, first of all, has the 'q' and 'qq' operators. As much as I'd
come to dislike Perl after I discovered Python, I miss those two.
Every time I have to quote a string full of single/double quotes,
? ? q{'this' is not this, but 'that' is 'that' like 'this'}
? ? q|'this' is not this, but 'that' is 'that' like 'this'|
? ? q<'this' is not this, but 'that' is 'that' like 'this'>
... with 'qq' providing the version with inerpolation. I could
always find an arbitrary character for quoting that was _not_ present
in the string, and so, most of the time, avoid quoting altogether.
It was perhaps a bit too magical, but pruced very readable strings.
Yes the q of perl is qute and qlever.
Come to think of it its very sensible because it factors delimition
from quoting just as the () and ' do for lisp.
rusi
2012-04-06 15:25:59 UTC
Permalink
On Apr 6, 6:55?pm, Steven D'Aprano <steve
Post by Steven D'Aprano
Post by rusi
Are there languages (other than python) in which single and double
quotes are equivalent?
Classic REXX, CSS, JavaScript, Lua, Prolog, XPath, YAML, Modula-2, HTML,
and (of course) English. There may be others.
Other languages like Perl, PHP and Ruby support alternate delimiters with
slightly different semantics.
--
Steven
Prolog: seems to be different, see
http://stackoverflow.com/questions/4391435/how-to-manipulate-strings-in-prolog

yaml: double seems to be like python triple, ie multi-line allowed
whereas single does not allow multi-line

http://yaml.org/spec/current.html#single%20quoted%20style/syntax
Barry W Brown
2012-04-07 00:48:41 UTC
Permalink
Post by rusi
Post by Duncan Booth
? ? v = json.loads("{'test':'test'}") ?fails
? ? v = json.loads('{"test":"test"}') ?succeeds
You mean JSON expects a string with valid JSON?
Quelle surprise.
Are there languages (other than python) in which single and double
quotes are equivalent?
Fortran 90 and more recent versions.
Post by rusi
[No I dont claim to know all the languages out there, just that I dont
know any other language which allows single and double quotes to be
interconvertible like python does]
Post by Duncan Booth
? ? v = json.loads("{'test':'test'}") ?fails
? ? v = json.loads('{"test":"test"}') ?succeeds
You mean JSON expects a string with valid JSON?
Quelle surprise.
Are there languages (other than python) in which single and double
quotes are equivalent?
[No I dont claim to know all the languages out there, just that I dont
know any other language which allows single and double quotes to be
interconvertible like python does]
Cameron Simpson
2012-04-04 23:18:55 UTC
Permalink
On 04Apr2012 15:34, Miki Tebeka <miki.tebeka at gmail.com> wrote:
| I'm going to give a "Python Gotcha's" talk at work.
| If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
|
| (Note that I want over http://wiki.python.org/moin/PythonWarts already).

Missing "return" means "return None".
One only has to miss a control path to have this happen.
Easy enough to accomodate, if only by having a return or raise at the
end of the function, but ...
--
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

I really don't like :-) symbols as well. I feel that if you can't see the pie
coming, you deserve whipped cream up your nose.
- robd at cherry.cray.com (rob derrick)
Cameron Simpson
2012-04-04 23:28:38 UTC
Permalink
@property
... def x():
... print 1
...
x
<property object at 0x100475f18>

Actually, that doesn't work with classes either, only instances.
Can I refer to the instance of the module/package?

In the interpreter, __package__ is None instead of some unnamed
"module/package".

In general I have a nagging desire that modules were more like classes.
But the details remain nebulous in my mind. The property thing is the
only concrete thing I trip over at present.
--
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Louis Pasteur's theory of germs is ridiculous fiction.
--Pierre Pachet, Professor of Physiology at Toulouse, 1872
Chris Angelico
2012-04-04 23:32:44 UTC
Permalink
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
Don't know if it's what's meant on that page by the += operator, but
the weirdness with concatenating onto a list inside a tuple may merit
a mention.
Post by Miki Tebeka
a=([1],)
a[0].append(2) # This is fine
a
([1, 2],)
Post by Miki Tebeka
a[0]+=[3] # This is not.
Traceback (most recent call last):
File "<pyshell#134>", line 1, in <module>
a[0]+=[3]
TypeError: 'tuple' object does not support item assignment
Post by Miki Tebeka
a
([1, 2, 3],)

Throws exception, but still does append to the list.

ChrisA
John Posner
2012-04-05 14:15:03 UTC
Permalink
Post by Chris Angelico
Don't know if it's what's meant on that page by the += operator,
Yes, it is.
Post by Chris Angelico
a=([1],)
a[0].append(2) # This is fine
[In the following, I use the term "name" rather loosely.]

The append() method attempts to modify the object whose name is "a[0]".
That object is a LIST, so the attempt succeeds.
Post by Chris Angelico
a[0]+=[3] # This is not.
The assignment attempts to modify the object whose name is "a". That
object is a TUPLE, so the attempt fails. This might be a surprise, but
I'm not sure it deserves to be called a wart.

Note the similarity to:

temp = a[0] + [3] # succeeds
a[0] = temp # fails

-John
Robert Kern
2012-04-05 14:45:57 UTC
Permalink
Post by John Posner
Post by Chris Angelico
Don't know if it's what's meant on that page by the += operator,
Yes, it is.
Post by Chris Angelico
a=([1],)
a[0].append(2) # This is fine
[In the following, I use the term "name" rather loosely.]
The append() method attempts to modify the object whose name is "a[0]".
That object is a LIST, so the attempt succeeds.
Post by Chris Angelico
a[0]+=[3] # This is not.
The assignment attempts to modify the object whose name is "a". That
object is a TUPLE, so the attempt fails. This might be a surprise, but
I'm not sure it deserves to be called a wart.
The wart is not that it fails, but that it does not fail atomically. The list
inside the tuple gets modified even though an exception is raised for the
statement as a whole.
--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
Michael Hrivnak
2012-04-05 18:44:16 UTC
Permalink
This is not a gotcha, and it's not surprising. As John described,
you're assigning a new value to an index of a tuple, which tuples
don't support.

a[0] += [3]

is the same as

a[0] = a[0] + [3]

which after evaluation is the same as

a[0] = [1, 3]

You can always modify an item that happens to be in a tuple if the
item itself is mutable, but you cannot add, remove, or replace items
in a tuple.

Michael
Post by John Posner
Post by Chris Angelico
Don't know if it's what's meant on that page by the += operator,
Yes, it is.
Post by Chris Angelico
a=([1],)
a[0].append(2) # This is fine
[In the following, I use the term "name" rather loosely.]
The append() method attempts to modify the object whose name is "a[0]".
That object is a LIST, so the attempt succeeds.
Post by Chris Angelico
a[0]+=[3] # This is not.
The assignment attempts to modify the object whose name is "a". That
object is a TUPLE, so the attempt fails. This might be a surprise, but
I'm not sure it deserves to be called a wart.
temp = a[0] + [3] ? # succeeds
a[0] = temp ? ? ? ? # fails
-John
--
http://mail.python.org/mailman/listinfo/python-list
Chris Angelico
2012-04-05 18:46:09 UTC
Permalink
This is not a gotcha, and it's not surprising. ?As John described,
you're assigning a new value to an index of a tuple, which tuples
don't support.
a[0] += [3]
is the same as
a[0] = a[0] + [3]
which after evaluation is the same as
a[0] = [1, 3]
You can always modify an item that happens to be in a tuple if the
item itself is mutable, but you cannot add, remove, or replace items
in a tuple.
It does make sense, and I've never actually had problems with it
myself. But it's come up on the list and clearly been a cause of
confusion for people, so I thought it worth mentioning. And, as it
turns out, it was the entry already on the list.

ChrisA
Evan Driscoll
2012-04-05 22:11:47 UTC
Permalink
Post by Michael Hrivnak
This is not a gotcha, and it's not surprising. As John described,
you're assigning a new value to an index of a tuple, which tuples
don't support.
Um, at least for me personally, yes, it is surprising, and yes, it is a
gotcha.


This goes back to what languages you're used to.

I come from a strong C++ background. In C++, something like mytuple[0]
+= [3] would wind up calling 'operator+=' on the list, that += would
mutate the list, and then the expression would be done.

The C++ approach doesn't really work with Python because more stuff is
immutable, and Python's behavior is arguably the least-bad solution, but
it WAS definitely surprising for me (and others!) when I learned about
Python's behavior.

In particular, the translation of 'a+=b' to 'temp = a + b; a = temp' is
*not* a very natural one to me.

Evan
Post by Michael Hrivnak
Michael
Post by John Posner
Don't know if it's what's meant on that page by the +=perator,
Yes, it is.
a=1],)
a[0].append(2) # This is fine
[In the following, I use the term "name" rather loosely.]
The append() method attempts to modify the object whose name is "a[0]".
That object is a LIST, so the attempt succeeds.
a[0]+=] # This is not.
The assignment attempts to modify the object whose name is "a". That
object is a TUPLE, so the attempt fails. This might be a surprise, but
I'm not sure it deserves to be called a wart.
temp =[0] + [3] # succeeds
a[0] =emp # fails
-John
--
http://mail.python.org/mailman/listinfo/python-list
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 552 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20120405/76bf4520/attachment.pgp>
Evan Driscoll
2012-04-05 22:19:41 UTC
Permalink
Post by Evan Driscoll
In particular, the translation of 'a+=b' to 'temp = a + b; a = temp' is
*not* a very natural one to me.
To expand on this point slightly, because of common C++ idioms guided by
efficiency, I would be much more likely to think of 'a + b' as 'temp =
a, temp += b, temp' than the Python way: in other words, to the extent
that I think of += and + being implemented in terms of each other, I
think of + as being implemented via +=.

Evan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 552 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20120405/bed76748/attachment.pgp>
John O'Hagan
2012-04-06 03:44:33 UTC
Permalink
On Thu, 05 Apr 2012 10:15:03 -0400
Post by John Posner
Post by Chris Angelico
Don't know if it's what's meant on that page by the += operator,
Yes, it is.
Post by Chris Angelico
a=([1],)
a[0].append(2) # This is fine
[In the following, I use the term "name" rather loosely.]
The append() method attempts to modify the object whose name is "a[0]".
That object is a LIST, so the attempt succeeds.
Post by Chris Angelico
a[0]+=[3] # This is not.
The assignment attempts to modify the object whose name is "a". That
object is a TUPLE, so the attempt fails. This might be a surprise, but
I'm not sure it deserves to be called a wart.
There was a thread called "copy on write" several weeks ago which veered into a
discussion of this:

http://mail.python.org/pipermail/python-list/2012-January/1286466.html

While I follow the reason for the exception noted above, to me this is a gotcha,
and has at least two wart-like features:

1) The mutation clearly intended by a[0] += [3] succeeds, even though an
exception is raised. What fails is the subsequent assignment, which makes no
difference in this case. I wouldn't blame anyone for being surprised to find
that a[0] is now [1, 2, 3] despite the exception.

2) Whether the operation succeeds depends on what name we use to refer to the
Post by John Posner
Post by Chris Angelico
a = ([],)
b = a[0]
b is a[0]
True
Post by John Posner
Post by Chris Angelico
b += [1]
a
([1],)
Post by John Posner
Post by Chris Angelico
a[0] += [2]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
Post by John Posner
Post by Chris Angelico
a
([1, 2],)
Post by John Posner
Post by Chris Angelico
b is a[0]
True

Very surprising to me, despite knowing why it happens. I used to believe that
how an object is referred to had no effect on its behaviour as an operand.

In the abovementioned thread, Hrvoje Niksic posted an implementation approach
which avoided all this by "simply not perform[ing] the final assignment if the
in-place method is available on the contained object":

http://mail.python.org/pipermail/python-list/2012-February/1287400.html

However, I get the impression this issue is generally regarded as "least worst,
won't fix".

Regards,

John
Tim Chase
2012-04-04 23:42:56 UTC
Permalink
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
1) While I believe it was fixed in more recent releases (perhaps
Py3 or later, most of my code is still in 2.x), leaking of
list-comprehension variables into the surrounding scope has stung
me on occasion:

val = something_important
whatever = [val for val in iterable if condition(val)]
assert val == something_important, "ug!"


2) While totally understandable, the significance of leading
whitespace in docstrings/triple-quoted items occasionally catches
me in places I didn't intentionally want it:

class Foo:
def frobniculate(self, x, y):
"""Frobniculate the x & y
x is the macrowobble variance
y is the miniwibble grobulation
"""
pass

(there's now leading whitespace on lines 2 & 3, and an extra
trailing line of pure whitespace).

3) the peculiarities of old-style classes and new-style classes
in 2.x (mooted by 3.x) take careful reading of the docs if you're
overriding __getattr__ or __getattribute__, as well as possible
other old-vs-new gotchas.

4) the __del__ method may have things in the containing scopes
(such as modules imported at the top of the containing module)
that get garbage-collected before the __del__ is called. In
general, __del__ stinks, and is better replaced by "with"
statements using context managers.

5) the masking of system modules. On multiple occasions I've
created a local module named "email.py", and then upon importing
something else (I think it may have been smtplib...not sure), it
failed with a confusing error because the imported module found
mine rather than the system module.

Those are my off-the-top-of-the-head gotchas.

-tkc
Iain King
2012-04-05 15:23:44 UTC
Permalink
A common one used to be expecting .sort() to return, rather than mutate (as it does). Same with .reverse() - sorted and reversed have this covered, not sure how common a gotcha it is any more.


Iain
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
Thanks,
--
Miki
Steve Howell
2012-04-05 15:35:37 UTC
Permalink
A common one used to be expecting .sort() to return, rather than mutate (as it does). ?Same with .reverse() - sorted and reversed have this covered, not sure how common a gotcha it is any more.
The sort()/sorted() variations are good to cover. To give another
example, folks who had been immersed in legacy versions of Python for
a long time might still be in the habit of hand-writing compare
functions. With newer versions of Python, it usually makes sense to
just use the "key" feature.
John Gordon
2012-04-05 16:12:16 UTC
Permalink
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
This is fairly pedestrian as gotchas go, but it has bitten me:

If you are working with data that is representable as either an integer
or a string, choose one and stick to it. Treating it as both/either will
eventually lead to grief.

Or, in other words: 1 != '1'
--
John Gordon A is for Amy, who fell down the stairs
gordon at panix.com B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"
Roy Smith
2012-04-05 18:08:13 UTC
Permalink
In article <jlkg90$ghk$1 at reader1.panix.com>,
In <7367295.815.1333578860181.JavaMail.geo-discussion-forums at ynpp8> Miki
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
If you are working with data that is representable as either an integer
or a string, choose one and stick to it. Treating it as both/either will
eventually lead to grief.
Or, in other words: 1 != '1'
Tell that to the PHP crowd :-)
Chris Angelico
2012-04-05 18:24:00 UTC
Permalink
Post by Roy Smith
Post by John Gordon
If you are working with data that is representable as either an integer
or a string, choose one and stick to it. ?Treating it as both/either will
eventually lead to grief.
Or, in other words: 1 != '1'
Tell that to the PHP crowd :-)
I think this example highlights a major point about gotchas: the
difference between an obvious language feature and a gotcha depends on
where you come from. To a PHP programmer, 1 and "1" are in many ways
indistinguishable. To a C programmer, they're utterly incompatible.

ChrisA
Evan Driscoll
2012-04-05 18:54:35 UTC
Permalink
Post by Chris Angelico
I think this example highlights a major point about gotchas: the
difference between an obvious language feature and a gotcha depends on
where you come from. To a PHP programmer, 1 and "1" are in many ways
indistinguishable. To a C programmer, they're utterly incompatible.
I think I agree with this. For instance, I'd consider the fact that this
Post by Chris Angelico
1 < "abc"
True

But that's because I like more strongly-typed systems. [I'm most
decidedly not trying to start up that discussion again...]
Post by Chris Angelico
1 < "abc"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < str()
Post by Chris Angelico
0 < True
True


I think it also has bearing on the ' vs " issue. For instance, I totally
think it's not at all surprising that one can be accepted and the other
not, or that they behave differently. (Though I *do* find it surprising
in the context of JSON given that JS apparently allows either.)

Evan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 552 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20120405/405c4b15/attachment.pgp>
Alain Ketterlin
2012-04-05 17:13:54 UTC
Permalink
Miki Tebeka <miki.tebeka at gmail.com> writes:

[...]
Post by Miki Tebeka
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
The "local variable and scoping" is, imho, something to be really
careful about. Here is an example:

class A(object):
def __init__(self):
self.x = 0
def r(self):
return x # forgot self

a = A()
x = 1
print a.r() # prints 1

I know there is "no remedy". It's just really tricky.

-- Alain.
Cameron Simpson
2012-04-06 00:02:35 UTC
Permalink
On 05Apr2012 19:13, Alain Ketterlin <alain at dpt-info.u-strasbg.fr> wrote:
| Miki Tebeka <miki.tebeka at gmail.com> writes:
| > (Note that I want over http://wiki.python.org/moin/PythonWarts already).
|
| The "local variable and scoping" is, imho, something to be really
| careful about. Here is an example:
|
| class A(object):
| def __init__(self):
| self.x = 0
| def r(self):
| return x # forgot self
|
| a = A()
| x = 1
| print a.r() # prints 1
|
| I know there is "no remedy". It's just really tricky.

Whoa!

I presume this jost happens with globals and not with one function
calling another... (I guess I should test that instead of asking such a
dumb question).
--
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

You see, wire telegraph is a kind of a very, very long cat. You pull his tail
in New York and his head is meowing in Los Angeles. Do you understand this?
And radio operates exactly the same way: you send signals here, they receive
them there. The only difference is that there is no cat.
- Albert Einstein, when asked to describe radio
Dave Angel
2012-04-06 01:38:37 UTC
Permalink
Post by Cameron Simpson
| > (Note that I want over http://wiki.python.org/moin/PythonWarts already).
|
| The "local variable and scoping" is, imho, something to be really
|
| self.x = 0
| return x # forgot self
|
| a = A()
| x = 1
| print a.r() # prints 1
|
| I know there is "no remedy". It's just really tricky.
Whoa!
I presume this jost happens with globals and not with one function
calling another... (I guess I should test that instead of asking such a
dumb question).
It doesn't generally happen "with one function calling another." What
it does apply to is to "variables" of nested scope. If the function
doesn't have a defining line for x, then the runtime looks one level
out. in this case, that's to globals. But in more complex programs, for
example with nested functions, it might be to other places.
--
DaveA
Michael Hrivnak
2012-04-05 18:52:00 UTC
Permalink
... stuff.append('bar')
... print stuff
...
Post by Miki Tebeka
foo()
['bar']
Post by Miki Tebeka
foo()
['bar', 'bar']
Post by Miki Tebeka
foo()
['bar', 'bar', 'bar']
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
Thanks,
--
Miki
--
http://mail.python.org/mailman/listinfo/python-list
Ian Kelly
2012-04-05 19:15:00 UTC
Permalink
The OP beat you to it -- it's in the list at the wiki link.
Jon Clements
2012-04-05 18:10:51 UTC
Permalink
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
Thanks,
--
Miki
One I've had to debug...
Post by Miki Tebeka
text = 'abcdef'
print 'found it!'
# Nothing prints as bool(0) is False
print 'found it!'
found it!

Someone new who hasn't read the docs might try this, but then I guess it's not really a gotcha if they haven't bothered doing that.
Emile van Sebille
2012-04-05 20:06:36 UTC
Permalink
On 4/5/2012 11:10 AM Jon Clements said...
Post by Jon Clements
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
Thanks,
--
Miki
One I've had to debug...
Post by Miki Tebeka
text = 'abcdef'
print 'found it!'
# Nothing prints as bool(0) is False
print 'found it!'
found it!
Someone new who hasn't read the docs might try this, but then I guess it's not really a gotcha if they haven't bothered doing that.
Kind of begs for a contains method that returns the appropriate boolean:

if text.contains('bob')

Emile
Arnaud Delobelle
2012-04-05 20:12:50 UTC
Permalink
Post by Emile van Sebille
if text.contains('bob')
It's already there:

text.__contains__('bob')

It's usually spelt otherwise though:

'bob' in text
--
Arnaud
Ian Kelly
2012-04-05 20:14:16 UTC
Permalink
Post by Emile van Sebille
On 4/5/2012 11:10 AM Jon Clements said...
Post by Jon Clements
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
Thanks,
--
Miki
One I've had to debug...
Post by Miki Tebeka
text = 'abcdef'
? ? ? ?print 'found it!'
# Nothing prints as bool(0) is False
? ? ? ?print 'found it!'
found it!
Someone new who hasn't read the docs might try this, but then I guess it's
not really a gotcha if they haven't bothered doing that.
if text.contains('bob')
You mean like this?

if 'bob' in text:
print 'found it!'

Cheers,
Ian
Chris Angelico
2012-04-05 23:00:37 UTC
Permalink
Post by Jon Clements
One I've had to debug...
text = 'abcdef'
? ? ? ?print 'found it!'
# Nothing prints as bool(0) is False
? ? ? ?print 'found it!'
found it!
Someone new who hasn't read the docs might try this, but then I guess it's not really a gotcha if they haven't bothered doing that.
But your original will actually never work, since find() returns -1 if
not found. This one is far more a gotcha in PHP, where the equivalent
function returns boolean False if not found, or index (starting from
0) if found. Returning -1 on not-found is far more logical. (Plus, of
course, the 'substring in string' syntax exists for just testing.)

ChrisA
John Nagle
2012-04-08 04:01:37 UTC
Permalink
Post by Miki Tebeka
Greetings,
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
Thanks,
--
Miki
A few Python "gotchas":

1. Nobody is really in charge of third party packages. In the
Perl world, there's a central repository, CPAN, and quality
control. Python's "pypi" is just a collection of links. Many
major packages are maintained by one person, and if they lose
interest, the package dies.

2. C extensions are closely tied to the exact version of CPython
you're using, and finding a properly built version may be difficult.

3. "eggs". The "distutils" system has certain assumptions built into
it about where things go, and tends to fail in obscure ways. There's
no uniform way to distribute a package.

4. The syntax for expression-IF is just weird.

5. "+" as concatenation. This leads to strange numerical
semantics, such as (1,2) + (3,4) is (1,2,3,4). But, for
"numarray" arrays, "+" does addition. What does a mixed
mode expression of a numarray and a tuple do? Guess.

5. It's really hard to tell what's messing with the
attributes of a class, since anything can store into
anything. This creates debugging problems.

6. Multiple inheritance is a mess. Especially "super".

7. Using attributes as dictionaries can backfire. The
syntax of attributes is limited. So turning XML or HTML
structures into Python objects creates problems.

8. Opening a URL can result in an unexpected prompt on
standard input if the URL has authentication. This can
stall servers.

9. Some libraries aren't thread-safe. Guess which ones.

10. Python 3 isn't upward compatible with Python 2.

John Nagle
Chris Angelico
2012-04-08 04:19:50 UTC
Permalink
4. ?The syntax for expression-IF is just weird.
Agreed. Putting an expression first feels weird; in every high level
language I know of, the word "if" is followed by the condition, and
then by what to do if true, and then what to do if false - not true,
then condition, then false.
6. ?Multiple inheritance is a mess. ?Especially "super".
Can you name any language in which multiple inheritance is NOT a mess?

Okay, so I'm a bit cynical. But MI is its own problem, and I think the
Python 3 implementation is about as good as it's worth hoping for.

ChrisA
Chris Angelico
2012-04-08 04:21:42 UTC
Permalink
Post by Chris Angelico
Agreed. Putting an expression first feels weird; in every high level
language I know of, the word "if" is followed by the condition, and
then by what to do if true, and then what to do if false - not true,
then condition, then false.
Clarification: I'm talking primarily about statement-if here. Not many
languages have an expression-if that isn't derived from either LISP or
C; both of those still have the three parts in the same order, so it
comes to the same thing. Python switches them around compared to that.

ChrisA
Roy Smith
2012-04-08 13:49:11 UTC
Permalink
In article <jlr2iu$ish$1 at dont-email.me>, John Nagle <nagle at animats.com>
Post by John Nagle
1. Nobody is really in charge of third party packages. In the
Perl world, there's a central repository, CPAN, and quality
control. Python's "pypi" is just a collection of links. Many
major packages are maintained by one person, and if they lose
interest, the package dies.
While I agree that this is a problem, it's not specifically a Python
problem. There's a lot of abandonware out there. In all languages.
With both OSS and commercial products.

Having an official curated central repository is a good thing, but it
has its down side too. What happens when the curator decides not to
allow your code into the library? Down that path lies things like the
Apple Store for IOS. If Apple decides they don't want your app for
whatever reason, your app is dead.
Miki Tebeka
2012-04-08 17:55:11 UTC
Permalink
Post by John Nagle
8. Opening a URL can result in an unexpected prompt on
standard input if the URL has authentication. This can
stall servers.
Can you give an example? I don't think anything in the standard library does that.
Chris Angelico
2012-04-08 18:24:42 UTC
Permalink
Post by Miki Tebeka
8. ?Opening a URL can result in an unexpected prompt on
standard input if the URL has authentication. ?This can
stall servers.
Can you give an example? I don't think anything in the standard library does that.
I just threw together a quick test with a page on my local server that
returns 401 Unauth and calls for basic authentication. With Python 2:

Python 2.4.5 (#1, Jul 22 2011, 02:01:04)
[GCC 4.1.1] on mingw32
Type "help", "copyright", "credits" or "license" for more information.
Post by Miki Tebeka
import urllib
urllib.urlopen("http://CENSORED")
Enter username for CENSORED at CENSORED:
Warning: Problem with getpass. Passwords may be echoed.
Enter password for in CENSORED at CENSORED:
<addinfourl at 11192172 whose fp = <socket._fileobject object at 0x00ADAC34>>

I hit enter twice and that's what it did. (The "Enter password for
in" presumably would be quoting back my username.)

Using the same URL with urllib2.urlopen() threw an HTTPError exception
citing the 401.

In Python 3, urllib.request.urlopen() throws HTTPError as above.

So it seems that this gotcha, while a very real problem, is only an
issue with a deprecated module on the old branch of Python. Unless
it's somewhere else that I couldn't find?

ChrisA
John Nagle
2012-04-08 19:05:27 UTC
Permalink
Post by Miki Tebeka
Post by John Nagle
8. Opening a URL can result in an unexpected prompt on
standard input if the URL has authentication. This can
stall servers.
Can you give an example? I don't think anything in the standard library does that.
It's in "urllib". See

http://docs.python.org/library/urllib.html

"When performing basic authentication, a FancyURLopener instance calls
its prompt_user_passwd() method. The default implementation asks the
users for the required information on the controlling terminal. A
subclass may override this method to support more appropriate behavior
if needed."

A related "gotcha" is knowing that "urllib" sucks and you should use
"urllib2".

John Nagle
Neil Cerutti
2012-04-09 12:21:39 UTC
Permalink
Post by John Nagle
6. Multiple inheritance is a mess. Especially "super".
Python allows you to get dirty. Super solves a messy problem.
Post by John Nagle
10. Python 3 isn't upward compatible with Python 2.
Even minor versions of Python are usually not forward compatible.
In the case of 2 to 3, more help and support than usual is
available: http://docs.python.org/dev/howto/pyporting.html
--
Neil Cerutti
Stefan Schwarzer
2012-04-09 20:49:37 UTC
Permalink
Hi Miki,
Post by Miki Tebeka
I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
(Note that I want over http://wiki.python.org/moin/PythonWarts already).
I gave a somewhat similar talk a while ago:

http://sschwarzer.com/download/robust_python_programs_europython2010.pdf

The following is a German version of the talk slides, but
covers a bit more since there was a longer time slot. Even
if you don't know German, you'll most likely understand what
I'm talking about by reading the code. :-)

http://sschwarzer.com/download/robustere_python_programme_clt2010_print.pdf

Stefan
Bryan
2012-04-14 22:47:54 UTC
Permalink
Post by Miki Tebeka
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors, and in some/many/most *nix implementations,
it likes to install in the same place. Python 3 is an improvement upon
Python 2, but Python went from, "sure... Python just works," to,
"well... that depends... which Python?"

I missed the 1 to 2 transition. I'm not exactly a Python newbie, but
Python 1.5.2 was dead an buried by the time I met the snake^H^H^H^H^H
group of daffy English k-ni-ghits.

We knew that there was no painless path from where we were to where we
saw to be better, and the Python community and our BDFL made many wise
decisions toward the Python 3 transition. This particular matter, we
botched. I am too late in speaking up, so I'm as much blame as anyone.
Something to keep in mind for Python 4.

-Bryan
Chris Angelico
2012-04-14 23:02:53 UTC
Permalink
On Sun, Apr 15, 2012 at 8:47 AM, Bryan
Post by Bryan
Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors, and in some/many/most *nix implementations,
it likes to install in the same place. Python 3 is an improvement upon
Python 2, but Python went from, "sure... Python just works," to,
"well... that depends... which Python?"
But that's true of many things. (Is it still normal for web hosts to
offer PHP4 and PHP5?) The new version introduces something
incompatible, and some code breaks. Python has some excellent
facilities for managing this breakage (eg future imports), but it's
still always possible for somebody's script to break in an upgrade,
which is why package maintainers won't normally auto-upgrade you from
(say) 2.6.6 to 2.7.2, just in case. Python 3 is simply a larger body
of incompatible changes.
Post by Bryan
I missed the 1 to 2 transition. I'm not exactly a Python newbie, but
Python 1.5.2 was dead an buried by the time I met the snake^H^H^H^H^H
group of daffy English k-ni-ghits.
Same here, but when I went poking around on one of my OS/2 servers, I
discovered a rather ancient Python - 1.5 or thereabouts - still
happily living there alongside the 2.6 or 2.7 that was the active
Python...

ChrisA
MRAB
2012-04-14 23:14:57 UTC
Permalink
Post by Bryan
Post by Miki Tebeka
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors, and in some/many/most *nix implementations,
it likes to install in the same place. Python 3 is an improvement upon
Python 2, but Python went from, "sure... Python just works," to,
"well... that depends... which Python?"
I missed the 1 to 2 transition. I'm not exactly a Python newbie, but
Python 1.5.2 was dead an buried by the time I met the snake^H^H^H^H^H
group of daffy English k-ni-ghits.
We knew that there was no painless path from where we were to where we
saw to be better, and the Python community and our BDFL made many wise
decisions toward the Python 3 transition. This particular matter, we
botched. I am too late in speaking up, so I'm as much blame as anyone.
Something to keep in mind for Python 4.
I could also mention the strip/lstrip/rstrip methods. They accept a
string argument which is treated as a _set_ of characters to be
stripped from the string (sets were a later addition). Newbies
sometimes wonder why, for example, "test.txt".strip(".txt") returns
"tes" and not "test". The transition to Python 3 would've been a good
time to change that, although if the argument could be a set of
multicharacter strings then the order in which they are stripped would
matter, so perhaps a tuple of strings, like with the startswith and
endswith methods, would've been better.

I didn't think of it until it was too late...
Steven D'Aprano
2012-04-15 01:23:04 UTC
Permalink
Post by Bryan
Post by Miki Tebeka
If you have an interesting/common "Gotcha" (warts/dark corners ...) please share.
Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors,
And so it should.

Python 2 and Python 3 are two dialects of the same language, with just a
very few minor changes in syntax (although a fair number of changes in
libraries). With a modicum of care, it is quite possible, even trivially
easy, to write useful code that is compatible with both Python 2 and 3.
Here's an example:

http://pypi.python.org/pypi/pyprimes
Post by Bryan
and in some/many/most *nix implementations,
it likes to install in the same place.
I won't speak for Unixes, but that is certainly not the case with Linux.
Each Python version gets its own location:

[steve at ando ~]$ for vers in 2.4 2.5 2.6 3.2 ; do which python$vers ; done
/usr/bin/python2.4
/usr/local/bin/python2.5
/usr/local/bin/python2.6
/usr/local/bin/python3.2


The first, 2.4, is the system python; the others were installed by me. I
didn't need to take any special effort to install to versioned locations,
the make script does that by default. The only special effort needed was
to run "make altinstall" instead of "make install" to ensure that
"python" without a version number still points to the system python
rather than the freshly installed version.
Post by Bryan
Python 3 is an improvement upon
Python 2, but Python went from, "sure... Python just works," to,
"well... that depends... which Python?"
I think that's nonsense. There has never been a time that you didn't have
to think about "which Python", except maybe the very first public
release. Every version has had differences from previous versions -- they
wouldn't be different versions otherwise. Bugs are fixed, but more
importantly new features are added. Libraries are deprecated and then
removed. If you write code using Python 2.4 features, it may not work
with Python 2.3. If you write code using 2.5 features like ternary if
expressions, it certainly will not work in Python 2.4.

The 2to3 transition isn't the only time that Python has made backwards
incompatible changes. I recently had to upgrade some Python code using
string exceptions from Python 2.3 to 2.6. That was tedious but easy.
However getting the tkinter code to upgrade was exciting and difficult.

Another example: the semantics of hex() and oct() have changed, probably
sometime around 2.4 or 2.5. A third: the introduction of nested scopes
back in 2.1. This changed the behaviour of any code using nested
functions, and was the inspiration for the introduction of __future__. A
fourth: turning None into a keyword in version 2.4.

I don't intent to be rude, but anyone who isn't a complete newbie to
programming but is surprised to the point of "gotcha" by version
compatibilities simply hasn't been paying attention.
--
Steven
Bryan
2012-04-15 09:23:27 UTC
Permalink
Post by Steven D'Aprano
Post by Bryan
Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors,
And so it should.
We disagree. Not surprising in a "gotcha's" thread.
Post by Steven D'Aprano
Post by Bryan
and in some/many/most *nix implementations,
it likes to install in the same place.
I won't speak for Unixes, but that is certainly not the case with Linux.
Yes, that was just silly of me to write that. All I want is a new
general convention for the most-likely-to-work invocation that won't
break with the change: "#!/usr/bin/env python" for Python 2 versus,
for example, "#!/usr/bin/env python3". Of course that's not an issue
of where python is installed, just a recommended naming convention.
Post by Steven D'Aprano
I don't intent to be rude, but anyone who isn't a complete newbie to
programming but is surprised to the point of "gotcha" by version
compatibilities simply hasn't been paying attention.
My perspective is simply different from yours. I'm not the one who
installs python on most of the boxes where I work or play. There's
little consistency, so I love conventions that usually work. I'd like
to advocate for Python 3, but the default install on Windows
commandeers the '.py' extension and breaks stuff that currently works.

Here's a discussion of the issue from late 2008. Amusingly, one of the
arguments for not changing the file extension was that Python 2 would
be gone in a few years.
http://www.velocityreviews.com/forums/t647251-running-python-2-and-python-3-on-the-same-machine.html

-Bryan
Chris Angelico
2012-04-15 09:41:27 UTC
Permalink
On Sun, Apr 15, 2012 at 7:23 PM, Bryan
Post by Bryan
Yes, that was just silly of me to write that. All I want is a new
general convention for the most-likely-to-work invocation that won't
break with the change: "#!/usr/bin/env python" for Python 2 versus,
for example, "#!/usr/bin/env python3". Of course that's not an issue
of where python is installed, just a recommended naming convention.
That's what happens if you altinstall python3 already.

Separate point though. Forgive me if I'm being obtuse, but cannot you
with a shebang specify the path to the Python interpreter you want?

#!/usr/local/bin/python3.3

versus

#!/usr/local/bin/python2.7

Isn't that kinda the point of a shebang instead of a simple keyword?

ChrisA
Mark Lawrence
2012-04-15 10:58:22 UTC
Permalink
Post by Bryan
My perspective is simply different from yours. I'm not the one who
installs python on most of the boxes where I work or play. There's
little consistency, so I love conventions that usually work. I'd like
to advocate for Python 3, but the default install on Windows
commandeers the '.py' extension and breaks stuff that currently works.
PEP397 discusses this issue, a solution is already available which works
a treat.
Post by Bryan
-Bryan
--
Cheers.

Mark Lawrence.
Steven D'Aprano
2012-04-15 15:17:19 UTC
Permalink
Post by Bryan
Post by Steven D'Aprano
Post by Bryan
Python 3(K) likes to use the same '.py' file extension as its
incompatible predecessors,
And so it should.
We disagree. Not surprising in a "gotcha's" thread.
Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file extension
for Python 3 is a good idea, you are keeping it to yourself.

Python and C are different languages. Python 2 and Python 3 are not, they
are the same language with only a few minor dialect differences.

There is a practical argument against separate file extensions: which
extension do you use for code intended to run with both Python 2 and 3?

We didn't need a new file extension for the transition between Python 2.5
(string exceptions are legal) and Python 2.6 (string exceptions cause a
SyntaxError exception). Nor did we need a new file extension for the
transition between Python 2.1 (nested functions behaved one way) and
Python 2.2 (nested functions behaved a different way). We certainly
didn't have a new file extension when the bastion or gopher modules were
removed from the standard library, backwards-incompatible changes if I've
ever seen one.

If there's a good argument in favour of separate file extensions for
Python 2 and Python 3 (one which doesn't also apply to, say, Python 2.5
and 2.6, or 3.1 and 3.2) I'm afraid I don't know it.
--
Steven
Curt
2012-04-15 15:30:39 UTC
Permalink
Post by Steven D'Aprano
Post by Bryan
We disagree. Not surprising in a "gotcha's" thread.
Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file extension
for Python 3 is a good idea, you are keeping it to yourself.
Didn't you trim his reasons, speaking of trimming (maybe they were
nonsensical, or poor, but it seems they were there before you made them
disappear)?
Steven D'Aprano
2012-04-15 19:59:36 UTC
Permalink
Post by Curt
Post by Steven D'Aprano
Post by Bryan
We disagree. Not surprising in a "gotcha's" thread.
Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file
extension for Python 3 is a good idea, you are keeping it to yourself.
Didn't you trim his reasons, speaking of trimming (maybe they were
nonsensical, or poor, but it seems they were there before you made them
disappear)?
No, you remember wrongly. Bryan states that the use of the same file
extension is a gotcha, but that's it.

[quote]
Python 3(K) likes to use the same '.py' file extension as
its incompatible predecessors

http://mail.python.org/pipermail/python-list/2012-April/1290909.html


Python 3 uses the same file extension as its incompatible predecessors
for the same reason that Python 2.6 uses the same file extension as its
incompatible predecessors, and 2.5 as as its incompatible predecessors,
and so on all the way back.

"Incompatible" is not a binary state, it is a matter of degree. Python
3.1 is less compatible to Python 2.7 than 2.7 is to 2.6, but the vast
bulk of the language is still identical and code supporting everything
from 2.5 to 3.2 in one code base is possible.

In my experience, it is MUCH easier to write code targeting versions 2.5
through 3.2 than it is to target 2.4 and 2.5 only, on account of how
impoverished 2.4 is compared to 2.5. (I once started a project to
backport useful 2.5 features to 2.4. I gave up because it was just too
painful.)
--
Steven
Bryan
2012-04-15 20:01:54 UTC
Permalink
Post by Steven D'Aprano
Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file extension
for Python 3 is a good idea, you are keeping it to yourself.
On Windows the file extension determines what executable opens the
file. Running both Python 2 and Python 3 on Windows is painful where
it doesn't need to be. I'd like to encourage my users to check out
Python 3, but installing it on Windows will take over the '.py'
extension and break stuff that currently works.

Incidentally, I'm not actually advocating for '.py3'. I'm advocating
for '.py4'.
Post by Steven D'Aprano
Python and C are different languages. Python 2 and Python 3 are not, they
are the same language with only a few minor dialect differences.
You could think of it as the same file extension in a different
dialect, but really the works/broken distinction is more important
than language/dialect.
Post by Steven D'Aprano
There is a practical argument against separate file extensions: which
extension do you use for code intended to run with both Python 2 and 3?
The file extension default should work with the the recommended
transition method, which is not dual-major-version code. Admittedly,
support for such code has improved.
Post by Steven D'Aprano
We didn't need a new file extension for the transition between Python 2.5
(string exceptions are legal) and Python 2.6 (string exceptions cause a
SyntaxError exception). Nor did we need a new file extension for the
transition between Python 2.1 (nested functions behaved one way) and
Python 2.2 (nested functions behaved a different way). We certainly
didn't have a new file extension when the bastion or gopher modules were
removed from the standard library, backwards-incompatible changes if I've
ever seen one.
Python's management of backwards compatibility for minor version has
worked pretty well, at least for me. Major version simply do not
attempt backward compatibility.

Your experience seems to be quite different from mine. I don't recall
a minor version upgrade ever giving me significant trouble with my own
code. The issue has been the external libraries upon which I depend,
and they've tended to catch up quickly, unlike what we're seeing with
Python 3.
Post by Steven D'Aprano
If there's a good argument in favour of separate file extensions for
Python 2 and Python 3 (one which doesn't also apply to, say, Python 2.5
and 2.6, or 3.1 and 3.2) I'm afraid I don't know it.
Because it would allow Windows user to play with Python 3 alongside
Python 2, while waiting for external libraries to catch up. Of course
they can, as I am, but the gotchas are really annoying. With minor
versions its not a big deal if most users simply wait to do an
upgrade.

-Bryan
Terry Reedy
2012-04-15 23:24:41 UTC
Permalink
Post by Bryan
On Windows the file extension determines what executable opens the
file. Running both Python 2 and Python 3 on Windows is painful where
it doesn't need to be. I'd like to encourage my users to check out
Python 3, but installing it on Windows will take over the '.py'
extension and break stuff that currently works.
There used to be an option to not do that, but it seems to have either
be removed or hidden. I admit that telling people to re-install 2.7 (or
whatever), just to recapture the file extension, is a nuisance. The new
PEP397 runner seems to be a better solution.
Post by Bryan
The file extension default should work with the the recommended
transition method,
There is no single recommended transition method at present.
Post by Bryan
which is not dual-major-version code.
The acceptance of irrelevant 'u' prefixes for strings in 3.3 is de facto
acceptance of that as *one* recommended method.
--
Terry Jan Reedy
Mark Lawrence
2012-04-15 23:28:47 UTC
Permalink
Post by Bryan
Post by Steven D'Aprano
Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file extension
for Python 3 is a good idea, you are keeping it to yourself.
On Windows the file extension determines what executable opens the
file. Running both Python 2 and Python 3 on Windows is painful where
it doesn't need to be. I'd like to encourage my users to check out
Python 3, but installing it on Windows will take over the '.py'
extension and break stuff that currently works.
Incidentally, I'm not actually advocating for '.py3'. I'm advocating
for '.py4'.
Post by Steven D'Aprano
Python and C are different languages. Python 2 and Python 3 are not, they
are the same language with only a few minor dialect differences.
You could think of it as the same file extension in a different
dialect, but really the works/broken distinction is more important
than language/dialect.
Post by Steven D'Aprano
There is a practical argument against separate file extensions: which
extension do you use for code intended to run with both Python 2 and 3?
The file extension default should work with the the recommended
transition method, which is not dual-major-version code. Admittedly,
support for such code has improved.
Post by Steven D'Aprano
We didn't need a new file extension for the transition between Python 2.5
(string exceptions are legal) and Python 2.6 (string exceptions cause a
SyntaxError exception). Nor did we need a new file extension for the
transition between Python 2.1 (nested functions behaved one way) and
Python 2.2 (nested functions behaved a different way). We certainly
didn't have a new file extension when the bastion or gopher modules were
removed from the standard library, backwards-incompatible changes if I've
ever seen one.
Python's management of backwards compatibility for minor version has
worked pretty well, at least for me. Major version simply do not
attempt backward compatibility.
Your experience seems to be quite different from mine. I don't recall
a minor version upgrade ever giving me significant trouble with my own
code. The issue has been the external libraries upon which I depend,
and they've tended to catch up quickly, unlike what we're seeing with
Python 3.
Post by Steven D'Aprano
If there's a good argument in favour of separate file extensions for
Python 2 and Python 3 (one which doesn't also apply to, say, Python 2.5
and 2.6, or 3.1 and 3.2) I'm afraid I don't know it.
Because it would allow Windows user to play with Python 3 alongside
Python 2, while waiting for external libraries to catch up. Of course
they can, as I am, but the gotchas are really annoying. With minor
versions its not a big deal if most users simply wait to do an
upgrade.
-Bryan
You can as I've already said do this but apparently you're more than
happy to ignore solutions that have been given to you. The
implementation of PEP397 refers. What more do you want?
--
Cheers.

Mark Lawrence.
David Robinow
2012-04-16 01:21:16 UTC
Permalink
On Sun, Apr 15, 2012 at 4:01 PM, Bryan
Post by Bryan
On Windows the file extension determines what executable opens the
file. Running both Python 2 and Python 3 on Windows is painful where
it doesn't need to be. I'd like to encourage my users to check out
Python 3, but installing it on Windows will take over the '.py'
extension and break stuff that currently works.
Hire a qualified Windows specialist.
Tell him to type:
FTYPE Python.File="C:\Python27\python.exe" :%1: %*
Pay him $200.00
Andrew Berg
2012-04-16 02:00:38 UTC
Permalink
Post by Bryan
I'd like to encourage my users to check out
Python 3, but installing it on Windows will take over the '.py'
extension and break stuff that currently works.
Have you tried telling your users to tell the installer not to do that?
IIRC, it's a simple checkbox option. I have a 64-bit 3.2 as my main, but
I also have a 64-bit 3.3, a 32-bit 3.2 and a 32-bit 2.7 installed, and
not once did any of the installers hijack the association.
Post by Bryan
There used to be an option to not do that, but it seems to have either
be removed or hidden.
Perhaps it only appears for "custom installations" (or whatever it's
called). I had no problem installing 3.3a1 and upgrading it to 3.3a2
while keeping 3.2.x as the default.
--
CPython 3.2.3 | Windows NT 6.1.7601.17990
Steven D'Aprano
2012-04-16 05:34:46 UTC
Permalink
Post by Steven D'Aprano
Yes, but I have reasons for disagreeing, which you trimmed out of your
response. If you have reasons for thinking that a separate file
extension for Python 3 is a good idea, you are keeping it to yourself.
On Windows the file extension determines what executable opens the file.
Running both Python 2 and Python 3 on Windows is painful where it
doesn't need to be.
And how is that different from any other two versions of Python?

Unless both versions include the same libraries (either standard modules
or third party), and your code is written to use only the lowest common
denominator functionality, you will have problems regardless of which
versions they are.

It is true that simple Python code using no external libraries has a
better chance of working without modification across multiple 2.x
versions than across the 2-3 transition, but that's not an absolute.
Going from 2.5 to 2.6 can cause breakage; on the other hand with care you
can support 2.6-3.2. With some extra effort, you can support 2.5 or even
older too.

(Sufficiently trivial, or careful, code can probably support any version
of Python from 1.4 to 3.2 inclusive. But it wouldn't be pleasant.)

The situation is worse on Windows, as Windows doesn't support hash-bang
syntax. But that is being looked at now:

http://www.python.org/dev/peps/pep-0397/

which should make supporting multiple Python versions much more pleasant
Real Soon Now.
--
Steven
alex23
2012-04-16 05:36:24 UTC
Permalink
On Apr 16, 3:34?pm, Steven D'Aprano <steve
Post by Steven D'Aprano
The situation is worse on Windows, as Windows doesn't support hash-bang
http://www.python.org/dev/peps/pep-0397/
which should make supporting multiple Python versions much more pleasant
Real Soon Now.
I keep hoping it will, but it's still in draft & I haven't seen much
motion lately :(
Paul Rubin
2012-04-16 06:21:42 UTC
Permalink
Post by Steven D'Aprano
Running both Python 2 and Python 3 on Windows is painful where it
doesn't need to be.
And how is that different from any other two versions of Python?
1. The backwards incompatibility between 2 and 3 is much more serious
than between 2.x and 2.(x-1).

2. There is not much reason to run multiple 2.x's on the same system.
Post by Steven D'Aprano
Going from 2.5 to 2.6 can cause breakage;
That should be considered a bug in 2.6, but in any case it doesn't come
into play that often. It's nothing like the 3.x changes.
Post by Steven D'Aprano
on the other hand with care you can support 2.6-3.2.
The idea of the incompatibilities introduced to 3.x is that people
should USE them, since they supposedly make the language better.
Therefore "with care" goes against their purpose. Also, "with care"
only applies to script you write yourself. If you download a script
from somewhere else that relies on 3.x, you have to modify it to work
under 2.x and vice versa. If you download both kinds of scripts you
have to modify some of them or else run two Pythons.
Bryan
2012-04-16 06:44:13 UTC
Permalink
Post by Steven D'Aprano
And how is that different from any other two versions of Python?
Python 3.0, also known as ?Python 3000? or ?Py3K?, is the first ever
*intentionally backwards incompatible* Python release. --GVR
Post by Steven D'Aprano
Unless both versions include the same libraries (either standard modules
or third party), and your code is written to use only the lowest common
denominator functionality, you will have problems regardless of which
versions they are.
*My* code is not what's holding me back. I *like* moving my code to
Python 3. What's holding me back is library availability, and from
various readings on the transition to 3, I gather that's what holding
many, perhaps most, users back.

The need to keep using 2.x should not stop users from installing and
starting to muck with 3.0. That's much more painful than it has to be
because they both use the same file extension.

--
--Bryan
rusi
2012-04-16 07:21:16 UTC
Permalink
On Apr 16, 11:44?am, Bryan <bryanjugglercryptograp... at yahoo.com>
Post by Bryan
Post by Steven D'Aprano
And how is that different from any other two versions of Python?
Python 3.0, also known as ?Python 3000? or ?Py3K?, is the first ever
*intentionally backwards incompatible* Python release. --GVR
Post by Steven D'Aprano
Unless both versions include the same libraries (either standard modules
or third party), and your code is written to use only the lowest common
denominator functionality, you will have problems regardless of which
versions they are.
*My* code is not what's holding me back. I *like* moving my code to
Python 3. What's holding me back is library availability, and from
various readings on the transition to 3, I gather that's what holding
many, perhaps most, users back.
The need to keep using 2.x should not stop users from installing and
starting to muck with 3.0. That's much more painful than it has to be
because they both use the same file extension.
There are two kinds of languages
1. The supercalifragilisticexpialidocious language that exists
entirely on my own computer (and perhaps a small band of devotees)
(for some value of 'my'). IOW a 'research' language
2. Those that just exist like the sun, moon and taxes (Fortran, Cobol
and C come to mind)
If the ISO-C committee decided that garbage collection should exist in
C, they can legislate it with comparable ease to legislating a change
in the sun's gravitation.

The problem with python is that it sits in between. It is still
alive, growing, learning etc and yet the installed base is so big that
incompatibilities cause real trouble.

Speaking more quantitatively: we can work out the cost/benefit ratio.

Assume that all the changes in py3k are 'good', they are benefits -- b
Costs can be single-cased (s) -- basically those that can be handled
by a 2to3 module
Those that are not so handle-able are the sorry case (multicase m)

And so the cost/benefit ratio is (s + n*m)/b (n is the no of python
users/uses 'out there')
IOW the irony: the success of python (large n) implies the failure of
py3k.

Of course not so ironic if one considers that Fortran, Cobol, C cant
be changed precisely because they are so successful

Continue reading on narkive:
Loading...