Again, I still suck at documentation, and my “tutorials” aren’t in-depth enough. So hopefully this covers all of the questions regarding using the django-sphinx module.
The first thing you’re going to need to do is install the Sphinx search software. You will be able to get this through http://www.sphinxsearch.com/, or probably even port or aptitude.
Configure Sphinx
Once you have successfully installed Sphinx you need to configure it. Follow the directions in on their website for the basic configuration, but most importantly, you need to configure a search index which can relate to one of your models.
Here is an example of an index from Curse’s File model, which let’s you search via name, description, and tags on a file. Please note, that “base” is a base source definition we created which has a few defaults which we use, but this is unrelated to your source definition.
source files_file_en : base
{
sql_query = \
SELECT files_file.id, files_file.name, files_data.description, files_file.tags as tag \
FROM files_file JOIN files_data \
ON files_file.id = files_data.file_id \
AND files_data.lang = 'en' \
AND files_file.visible = 1 \
GROUP BY files_file.id
sql_query_info = SELECT * FROM files_file WHERE id=$id
}
Now that you have your source defined, you need to build an index which uses this source. I do recommend placing all of your sphinx information somewhere else, maybe /var/sphinx/data.
index files_file_en
{
source = files_file_en
path = /var/data/files_file_en
docinfo = extern
morphology = none
stopwords =
min_word_len = 2
charset_type = sbcs
min_prefix_len = 0
min_infix_len = 0
}
Configure Django
Now that you’ve configured your search index you need to setup the configuration for Django. The first step to doing this is to install the django-sphinx wrapper. First things first, download the zip archive, or checkout the source from http://code.google.com/p/django-sphinx/.
Once you have your files on the local computer or server, you can simple do sudo python setup.py install to install the library.
After installation you need to edit a few settings in settings.py, which, again, being that I suck at documentation, isn’t posted on the website.
The two settings you need to add are these:
SPHINX_SERVER = 'localhost' SPHINX_PORT = 3312
Setup Your Model
Now you are fully able to utilize Sphinx within Django. The next step is to actually attach your search index to a model. To do this, you will need to import djangosphinx and then attach the manager to a model. See the example below:
from django.db import models import djangosphinx class File(models.model): name = models.CharField() tags = models.CharField() # We actually store tags for efficiency in tag,tag,tag format here objects = models.Manager() search = djangosphinx.SphinxSearch(index="files_file_en")
The index argument is optional, and there are several other parameters you can pass, but you’ll have to look in the code (or pydoc if I did it right, but probably not).
Once we’ve defined the search manager on our model, we can access it via Model.manager_name and pass it many things like we could with a normal object manager in Django. The typical usage is Model.search.query('my fulltext query') which would then query sphinx, grab a list of IDs, and then do a Model.objects.filter(pk__in=[list of ids]) and return this result set.
Search Methods
There are a few additional methods which you can use on your search queryset besides the default query method. order_by, filter, count, and exclude to name a few. These don’t *quite* work the same as Django’s as they’re used directly within the search wrapper. So here’s a brief rundown of these:
query
This is your basic full-text search query. It works exactly the same as passing your query to the full-text engine. It’s search type will be based on the search mode, which, by default, is SPH_MATCH_EXTENDED.filter/exclude
The filter and excludes method holds the same idea as the normal queryset methods, except that it is used directly in Sphinx. What this means, is that you can only filter on attribute fields that are present in your search index.order_by
The order_by method also passes its parameters to Sphinx, with one exception. There are four reserved keywords:@id,@weight,@rank, and@relevance. These are detailed in the Sphinx documentation.select_related
This method is directly passed onto the Django queryset and holds no value to Sphinx.index_on
Allows you to specify which index(es) you are querying for. To query for multiple indexes you need to include a “content_type” name in your fields.
14 Responses to "In-Depth django-sphinx Tutorial"
Brilliant - thank you!
I now have Sphinx querying a table and returning ORM models.
One question - could this work for models that have other models as many-many attributes?
Oh - wait a minute… it doesn’t matter does it? because your code gets the models ‘proper’ from the ORM based on the ID sphinx returns… is that right?
thanks very much for the guide!!!!!
Hi, thanks for an article. Very helpful.
Where should I put my sphinx.conf file?
Is this method any better/worse/different than using the fulltext search in mysql?
Hi, any info in this vs. lucene?
Also, is truly easy to install? I get a lot of problems with pyLucene & xapian.
Also, work on solaris? i jost on joyent so is a must…
I believe Sphinx is MySQL only.
[…] In-Depth django-sphinx Tutorial | David Cramer.net Again, I still suck at documentation, and my “tutorials” aren’t in-depth enough. So hopefully this covers all of the questions regarding using the django-sphinx module. […]
[…] In-Depth django-sphinx Tutorial […]
Thanks for writing this post. I have been wanting to try Sphinx lately but I didn’t know how to set everything up. This post helped me out a lot.
El Excellente. =)
http://www.sphinxsearch.com/doc.html#quick-tour <– got it running in 15 minutes with this.
Those with simpler searching needs might check out http://www.mercurytide.co.uk/whitepapers/django-full-text-search/
Sphinx works with Mysql and Postgres, just remember to run configure with the –with-pgsql option.
Seems the search command line tools doesn’t like postgres though..
Won’t “Model.objects.filter(pk__in=[list of ids])” turn pathological with a really huge list of IDs? what if it returns several million, and you only want the first ten…it might be good to add a way to query for one id at a time, though I admit this would require rewriting all the Manager methods to filter individual objects.
wingedsubmariner,
The only gets executed when you slice the Search queryset. So when you do mymodel.search.query(’q')[0:10] that list of ids is only 10 long.
that was asome
Leave A Reply