Feature Requests item #617979, was opened at 2002-10-03 10:00 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=617979&group_id=5470

Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Michael Hudson (mwh) Assigned to: M.-A. Lemburg (lemburg) Summary: need easy way of decoding a literal

Initial Comment: Especially since PEP 263, we badly need a way of turning a Python string literal into the appropriate object.

The immediate need is to get the compiler package working again, but I'm sure there are other places it could be used.

I thought Martin already had a patch somewhere doing this, but I can't find it. Maybe he remembers?

----------------------------------------------------------------------

Date: 2004-03-29 12:26

Message: Logged In: YES user_id=6656

I'd have to admit to not being sure what I'm actually asking for in this report, so I won't object if it's closed.

I do think that, e.g., using the tokenize module has gotten harder since PEP 263, but I don't have a pressing problem right now...

----------------------------------------------------------------------

Comment By: Martin v. L?wis (loewis) Date: 2004-03-25 19:35

Message: Logged In: YES user_id=21627

It still needs to be implemented.

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 19:32

Message: Logged In: YES user_id=38388

Has there been any progress on this or can I close it ?

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh) Date: 2003-01-16 13:28

Message: Logged In: YES user_id=6656

For a bunch of reasons, I now care less about this so I'm going to reduce the priority.

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg) Date: 2002-10-04 12:15

Message: Logged In: YES user_id=38388

MWH: My first paragraph was refering to Martin's suggestion to use eval() for mapping literal strings to objects.

Please take this to the python-dev mailing list.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh) Date: 2002-10-04 10:00

Message: Logged In: YES user_id=6656

MAL: I don't understand your first paragraph at all.

The {sys.,}source_encoding variable might work, but it would be a dreadful hack for my purposes. Using eval() is already a nasty hack, IMHO.

I agree a mailing list is a more civilised interface than a web browser...

----------------------------------------------------------------------

Comment By: Martin v. L?wis (loewis) Date: 2002-10-03 20:21

Message: Logged In: YES user_id=21627

That would not be correct. If you have a (plain) string literal, it would end up encoded as UTF-8, whereas the original source encoding was, say, Latin-1. People will be confused if string literals don't come out in the expected encoding in interactive mode.

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg) Date: 2002-10-03 19:11

Message: Logged In: YES user_id=38388

I believe the correct way to handle this would be to have eval() accept Unicode and then convert it to UTF-8 for the parser to process. The PEP only talks about compile() which should accept Unicode objects, but eval() is certainly a candidate too.

About the source_encoding idea: wouldn't it be better to define this for interactive use in the sys module ? A Python startup script could then be used to set this variable in interactive mode.

BTW, it's better to discuss these things on python-dev than in these small textareas... IMHO, at least.

----------------------------------------------------------------------

Comment By: Martin v. L?wis (loewis) Date: 2002-10-03 18:55

Message: Logged In: YES user_id=21627

I see.

For a different reason (interactive mode), I have been contemplating to recognize a variable source_encoding in globals(), which would be set in interactive mode to the terminal's encoding, so that input of non-ASCII Unicode literals would "work", without having to type an encoding declaration each time.

I believe such a mechanism could also solve this problem; callers of eval would have to set the global.

What do you think?

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh) Date: 2002-10-03 17:45

Message: Logged In: YES user_id=6656

How do you get eval to take notice of the declared encoding?

What I want is the functionality of compile.c:parsestr available from Python.

----------------------------------------------------------------------

Comment By: Martin v. L?wis (loewis) Date: 2002-10-03 17:35

Message: Logged In: YES user_id=21627

Not sure what you are asking. eval() would certainly be an easy way, no? In any case, I'm not aware of an alternative.

----------------------------------------------------------------------