[sqlalchemy] Re: sqlalchemy object serialization / deserialization - Rick Morrison - com.googlegroups.sqlalchemy

7 messages in com.googlegroups.sqlalchemy[sqlalchemy] Re: sqlalchemy object se...

From	Sent On	Attachments
Matt	20 Dec 2007 16:48
Rick Morrison	20 Dec 2007 17:03
Matt	20 Dec 2007 17:35
Rick Morrison	20 Dec 2007 17:45
Lele Gaifax	21 Dec 2007 01:16
Michael Schlenker	21 Dec 2007 04:35
Michael Schlenker	21 Dec 2007 04:41

Subject:	[sqlalchemy] Re: sqlalchemy object serialization / deserialization
From:	Rick Morrison (rick...@gmail.com)
Date:	12/20/2007 05:45:23 PM
List:	com.googlegroups.sqlalchemy

Hey Matt.

Class hinting was a json-rpc 1.0 feature, subsequently dropped in 1.1. But that's a nit.

The real problem, though is reconstituing those hints -- the machinery for reinstantiating those objects has to come from somewhere, and that somewhere is going to your code -- there is no magic "just add water and reconstitute" my object built into Python.

If you really want to get into writing your own deserialization, you're already heading in the right direction: simplejson and cjson (and maybe others) -- have hooks to plug in your own serialization and deserialization code. I've used them both to do something almost exactly like this to serialize and deserialize JS dates, which normally can't be sent in JSON. You'll find that the built-in serialization works for *most* Python objects, especially for basic types like str(), int(), etc., but most certainly not *all* Python objects.

I don't really want to store JSON in the DB, but just use it as a serialization format for sqlalchemy objects. We want our frontend to render data from the same type of object with a couple different possible backend sources. One being the database through the sqlalchemy ORM and another being some sort of JSON/XML interface that we can backend from whatever...

Your best bet is going to be to deserialize the JSON to some intermediate object and then construct the SqlAlchemy instance. Re-implementing pickle with a JSON storage format is going to be a huge job, and really won't give you anything that you don't already get from pickle -- what other app besides you own is ever going to understand those heavily class-hinted JSON files?

On 12/20/07, Matt <ma...@vazor.com> wrote:

On Dec 20, 5:04pm, "Rick Morrison" <rick...@gmail.com> wrote:

You're not going to be able to serialize Python class instances in JSON: json strings are simple object literals limited to basic Javascript types. Pickle does some pretty heavy lifting to serialize and reconstitute class instances.

I've already figured this out -- you can use class "hinting" as outlined in this page:

http://json-rpc.org/wiki/specification

So a datetime using this format and converted to JSON might look like:

"modification_time": {"__jsonclass__": ["datetime.datetime", [2007, 12, 19, 23, 3, 2, 2]]}

My deserialization routine loads the datetime module, then gets the datetime attribute (which in this case is a type, but could be a function too), and calls that with the arguments in the list. This sort of thing works for generic classes too using the pickle __reduce__ hooks. The JSON part I'm actually handling through cjson or simplejson -- my serialization/deserialization is working all on python objects.

Easiest way to store JSON in the database is to limit the type of data you store in JSON strings to straightforward objects that only use primitive JS types, and then serialize back and forth to Python dictionaries. That's what libraries like SimpleJSON or cJSON do. Using Sqlalchemy, you can then store those JSON strings in database VARCHARS, TEXT and so on fields.