Comments on: Custom Fields in Django

By: AP

AP — Fri, 15 Jan 2010 18:34:37 +0000

Hi and thanks for this post. Let me bring up one issue with SerializedDataField which I suspect is a bug. I have a model class A with a serialized field s and a class B with a foreign key to A called a. When I do a A.objects.get(pk = …).s it works just fine but when I do a B.object.get.filter(…).values(‘a__s’) I get the raw base64-ed pickled string. I am sorry my example is not executable but the real thing is a bit more complicated than that. Let me know if you are looking into this and I will try and make a real example if that helps.

By: Custom Fields - help me, plz! )))

Custom Fields - help me, plz! ))) — Mon, 27 Jul 2009 17:42:00 +0000

[...] 21:41 ????? ?? ????? ?? ???????????? ????? ??? ?????????? ????, ??????????? ??????????? ??? ?? [...]

By: David Holstius

David Holstius — Fri, 07 Nov 2008 18:12:45 +0000

I'm running into the problem you documented, where Django's enthusiasm for to_python() means that this doesn't (seem to) work for Fields intended to handle strings.

We're trying to write a legacy-compatible Django app that can run alongside a legacy, non-unicode-aware PHP app. MySQL has the legacy tables as 'latin1' and the PHP is writing and reading UTF-8 bytestrings as payload in the textual columns.

One option we're tossing around is to change settings.DATABASE_OPTIONS['use_unicode'] to False, which means we get back the same bytestrings in Django, which we can then .decode('utf8'), but we're leery of this.

I came across this trick that will un-munge the munged unicode strings we get back if we leave 'use_unicode' set to True: value = munged_string.encode('iso-8859-1').decode('utf8'). Oof.

So maybe we can put this trick in to_python()---and put the converse in get_db_prep_value()---in a custom Field. But to_python seems to be called more than once, as you say.

Any advice?

P.S. Thanks for your djangosphinx module. Kudos. We're using that too, in the rewrite of of our site. It's positively a lifesaver.

By: David

David — Fri, 22 Aug 2008 16:43:25 +0000

For future readers, the above code *is* working.

By: David

David — Mon, 11 Aug 2008 13:19:43 +0000

I doubt it will go into trunk, but the above code wasn’t quite working. I’ve found more issues with it, due to Django’s custom field classes not being very well documented.

By: james_027

james_027 — Mon, 11 Aug 2008 12:59:41 +0000

hi,

this is very nice. You said “So I quickly did, and replaced a few lines of repetitive code with two new field classes in our source:” Does this mean that this will get into Django?

By: David

David — Fri, 08 Aug 2008 15:47:07 +0000

Ahh, seems I don’t know base64 as well as I thought. Thanks David!

We’ll just have to hope Django’s example of checking the value first is a waste of time.

By: David K

David K — Fri, 08 Aug 2008 15:45:10 +0000

Perhaps I’m missing something obvious, but I fail to understand the need to check for an ending “=” char in the encoded value before decoding and unpickling.

For instance, consider:

>>> import base64
>>> import cPickle as pickle
>>> base64.b64encode(pickle.dumps(['t','e','s','t']))
‘KGxwMQpTJ3QnCmFTJ2UnCmFTJ3MnCmFTJ3QnCmEu’

Section 2.2 of RFC 3548 seems to indicate that the ‘=’ padding is not required.