[google-appengine] Re: Unicode and the DataStore - Calvin Spealman - com.googlegroups.google-appengine

9 messages in com.googlegroups.google-appengine[google-appengine] Re: Unicode and th...

From	Sent On	Attachments
Rob	29 Jul 2008 14:03
Calvin Spealman	29 Jul 2008 14:37
Rob	30 Jul 2008 09:59
Calvin Spealman	30 Jul 2008 10:22
Rob	30 Jul 2008 22:30
Calvin Spealman	31 Jul 2008 07:58
Bruno Patini Furtado	31 Jul 2008 10:41
Calvin Spealman	31 Jul 2008 10:56
Jonathan Feinberg	01 Aug 2008 14:37

Subject:	[google-appengine] Re: Unicode and the DataStore
From:	Calvin Spealman (iron...@gmail.com)
Date:	07/30/2008 10:22:16 AM
List:	com.googlegroups.google-appengine

str objects are basically bytestrings. unicode objects are strings of unicode code points. StringProperties are trying to store them as text, which is best represented by a unicode object, not a str object. You could have a str object containing bytes that happen to be encoded text, but any functions you pass it around to can't be sure what encoding they are. The default is to assume any bytestring contains ascii compatible text and anything out of that range causes failure. In those failure cases, if you know the encoding of the text represented in the str, you should decode it into a unicode object as soon as possible. As a unicode object, python functions know its text, not bytes. And, if it gets passed to something expecting text, like in this case, it doesn't have to try and guess at the encoding, and possibly get it wrong, which is what it is doing.

unicode_string = str_string.decode('utf8')

That will decode the bytes in your str and produce a unicode object representing the proper text characters represented within the bytes. If you set the property to this, the datastore api doesn't need to attempt to decode it first, which is where your problem is happening (because it tries to decode the str into a unicode object for storage, but it doesn't know what encoding and it breaks, not being valid ASCII)

Does the distinction make sense now?

On Wed, Jul 30, 2008 at 1:00 PM, Rob <rob....@gmail.com> wrote:

I am very new to Python so I am not sure what you mean by a unicode object. devname = self.request.get('devname') According to the Datastore docs the db.StringProperty supports str and unicode.

On Jul 29, 2:38 pm, "Calvin Spealman" <iron...@gmail.com> wrote:

you probably have a str that contains non-ascii characters and you're trying to set that as the property value. If you have unicode text, you need to decode the str into a unicode object and save that, so db knows its a unicode string. str instances are otherwise assumed to be ASCII with all codes under 128.

On Tue, Jul 29, 2008 at 5:03 PM, Rob <rob....@gmail.com> wrote:

Hi All,

I am trying to save a value into my datastore the devName field below: class MyUsers(db.Model): user = db.UserProperty() devAddress = db.StringProperty(multiline=False) devName = db.StringProperty(multiline=False) date = db.DateTimeProperty(auto_now_add=True)

The value is user entered, and I am testing this on a Macintosh where the string contains a "curly single quote". I am URL-encoding the value as UTF-8, but the server throws the following error: pbvalue.set_stringvalue(unicode(v).encode('utf-8')) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 15: ordinal not in range(128)

The string should already be in UTF-8, it looks like it's trying to convert UTF8 to UTF8.

Any Clues?

Thanks, -Rob

- Show quoted text -