24

Feb

Filed in Code, Django, How-To's, Python with 14 comments |

Again, I still suck at documentation, and my “tutorials” aren’t in-depth enough. So hopefully this covers all of the questions regarding using the django-sphinx module.

The first thing you’re going to need to do is install the Sphinx search software. You will be able to get this through http://www.sphinxsearch.com/, or probably even port or aptitude.

Configure Sphinx

Once you have successfully installed Sphinx you need to configure it. Follow the directions in on their website for the basic configuration, but most importantly, you need to configure a search index which can relate to one of your models.

Here is an example of an index from Curse’s File model, which let’s you search via name, description, and tags on a file. Please note, that “base” is a base source definition we created which has a few defaults which we use, but this is unrelated to your source definition.

source files_file_en : base
{
sql_query			= \
	SELECT files_file.id, files_file.name, files_data.description, files_file.tags as tag \
	FROM files_file JOIN files_data \
	ON files_file.id = files_data.file_id \
	AND files_data.lang = 'en' \
	AND files_file.visible = 1 \
	GROUP BY files_file.id
sql_query_info		= SELECT * FROM files_file WHERE id=$id
}

Now that you have your source defined, you need to build an index which uses this source. I do recommend placing all of your sphinx information somewhere else, maybe /var/sphinx/data.

index files_file_en
{
	source			= files_file_en
	path			= /var/data/files_file_en
	docinfo			= extern
	morphology			= none
	stopwords			=
	min_word_len		= 2
	charset_type		= sbcs
	min_prefix_len		= 0
	min_infix_len		= 0
}

Configure Django

Now that you’ve configured your search index you need to setup the configuration for Django. The first step to doing this is to install the django-sphinx wrapper. First things first, download the zip archive, or checkout the source from http://code.google.com/p/django-sphinx/.

Once you have your files on the local computer or server, you can simple do sudo python setup.py install to install the library.

After installation you need to edit a few settings in settings.py, which, again, being that I suck at documentation, isn’t posted on the website.

The two settings you need to add are these:

SPHINX_SERVER = 'localhost'
SPHINX_PORT = 3312

Setup Your Model

Now you are fully able to utilize Sphinx within Django. The next step is to actually attach your search index to a model. To do this, you will need to import djangosphinx and then attach the manager to a model. See the example below:

from django.db import models
import djangosphinx
 
class File(models.model):
    name = models.CharField()
    tags = models.CharField() # We actually store tags for efficiency in tag,tag,tag format here
 
    objects = models.Manager()
    search  = djangosphinx.SphinxSearch(index="files_file_en")

The index argument is optional, and there are several other parameters you can pass, but you’ll have to look in the code (or pydoc if I did it right, but probably not).

Once we’ve defined the search manager on our model, we can access it via Model.manager_name and pass it many things like we could with a normal object manager in Django. The typical usage is Model.search.query('my fulltext query') which would then query sphinx, grab a list of IDs, and then do a Model.objects.filter(pk__in=[list of ids]) and return this result set.

Search Methods

There are a few additional methods which you can use on your search queryset besides the default query method. order_by, filter, count, and exclude to name a few. These don’t *quite* work the same as Django’s as they’re used directly within the search wrapper. So here’s a brief rundown of these:

  • query
    This is your basic full-text search query. It works exactly the same as passing your query to the full-text engine. It’s search type will be based on the search mode, which, by default, is SPH_MATCH_EXTENDED.
  • filter/exclude
    The filter and excludes method holds the same idea as the normal queryset methods, except that it is used directly in Sphinx. What this means, is that you can only filter on attribute fields that are present in your search index.
  • order_by
    The order_by method also passes its parameters to Sphinx, with one exception. There are four reserved keywords: @id, @weight, @rank, and @relevance. These are detailed in the Sphinx documentation.
  • select_related
    This method is directly passed onto the Django queryset and holds no value to Sphinx.
  • index_on
    Allows you to specify which index(es) you are querying for. To query for multiple indexes you need to include a “content_type” name in your fields.

14 Responses to "In-Depth django-sphinx Tutorial"

Subscribe to this topic with RSS or get the Trackback URL
Rich (Feb 25th):

Brilliant - thank you!
I now have Sphinx querying a table and returning ORM models.

One question - could this work for models that have other models as many-many attributes?

Oh - wait a minute… it doesn’t matter does it? because your code gets the models ‘proper’ from the ORM based on the ID sphinx returns… is that right?

thanks very much for the guide!!!!!

AndrewSK (Feb 25th):

Hi, thanks for an article. Very helpful.
Where should I put my sphinx.conf file?

john (Mar 6th):

Is this method any better/worse/different than using the fulltext search in mysql?

mamcx (Mar 6th):

Hi, any info in this vs. lucene?

Also, is truly easy to install? I get a lot of problems with pyLucene & xapian.

Also, work on solaris? i jost on joyent so is a must…

David (Mar 6th):

I believe Sphinx is MySQL only.

[…] In-Depth django-sphinx Tutorial | David Cramer.net Again, I still suck at documentation, and my “tutorials” aren’t in-depth enough. So hopefully this covers all of the questions regarding using the django-sphinx module. […]

[…] In-Depth django-sphinx Tutorial […]

Ryochan7 (Mar 24th):

Thanks for writing this post. I have been wanting to try Sphinx lately but I didn’t know how to set everything up. This post helped me out a lot.

Yeago (Apr 3rd):

El Excellente. =)

Yeago (Apr 5th):

http://www.sphinxsearch.com/doc.html#quick-tour <– got it running in 15 minutes with this.

Those with simpler searching needs might check out http://www.mercurytide.co.uk/whitepapers/django-full-text-search/

matt (Apr 7th):

Sphinx works with Mysql and Postgres, just remember to run configure with the –with-pgsql option.

Seems the search command line tools doesn’t like postgres though..

wingedsubmariner (Apr 11th):

Won’t “Model.objects.filter(pk__in=[list of ids])” turn pathological with a really huge list of IDs? what if it returns several million, and you only want the first ten…it might be good to add a way to query for one id at a time, though I admit this would require rewriting all the Manager methods to filter individual objects.

David (Apr 11th):

wingedsubmariner,

The only gets executed when you slice the Search queryset. So when you do mymodel.search.query(’q')[0:10] that list of ids is only 10 long.

jacob (May 5th):

that was asome

Leave A Reply

 Username (*required)

 Email Address (*private)

 Website (*optional)

Note: Comments moderation may be active so there is no need to resubmit your comment.