Django Blog Tutorial - the Next Generation - Part 7
Published by Matthew Daly at 25th August 2014 4:15 pm
Hello once again! In this instalment we'll cover:
- Caching your content with Memcached to improve your site's performance
- Refactoring and simplifying our tests
- Implementing additional feeds
- Creating a simple search engine
Don't forget to activate your virtualenv:
$ source venv/bin/activate
Now let's get started!
Memcached
If you frequent (or used to frequent) social media sites like Reddit, Slashdot or Digg, you may be familiar with something called variously the Digg or Slashdot effect, whereby if a page gets submitted to a social media site, and subsequently becomes popular, it can be hit by a huge number of HTTP requests in a very short period of time, slowing it down or even taking the server down completely.
Now, as a general rule of thumb, for most dynamic websites such as blogs, the bottleneck is not the web server or the programming language, but the database. If you have a lot of people hitting the same page over and over again in quick succession, then you're essentially running the same query over and over again and getting the same result each time, which is expensive in terms of processing power. What you need to be able to do is cache the results of the query in memory for a given period of time so that the number of queries is reduced.
That's where Memcached comes in. It's a simple key-value store that allows you to store values in memory for a given period of time so that they can be retrieved without having to query the database. Memcached is a very common choice for caching, and is by far the fastest and most efficient type of cache available for Django. It's also available on Heroku's free tier.
Django has a very powerful caching framework that supports numerous types of cache in addition to Memcached, such as:
- Database caching
- Filesystem caching
- Local memory caching
There are also third-party backends for using other caching systems such as Redis.
Now, the cache can be used in a number of different ways. You can cache only certain parts of your site if you wish. However, because our site is heavily content-driven, we should be pretty safe to use the per-site cache, which is the simplest way to set up caching.
In order to set up Memcached, there's a couple of Python libraries we'll need. If you want to install them locally, however, you'll need to install both memcached and libmemcached (on Ubuntu, the packages you need are called memcached and libmemcached-dev) on your development machine. If you don't want to do this, then just copy and paste these lines into requirements.txt instead:
1django-pylibmc-sasl==0.2.42pylibmc==1.3.0
If you are happy to install these dependencies locally, then run this command once memcached and libmemcached are installed:
$ pip install pylibmc django-pylibmc-sasl
With that done let's configure Memcached. Open up the settings file and add the following at the bottom:
1def get_cache():2 import os3 try:4 os.environ['MEMCACHE_SERVERS'] = os.environ['MEMCACHIER_SERVERS'].replace(',', ';')5 os.environ['MEMCACHE_USERNAME'] = os.environ['MEMCACHIER_USERNAME']6 os.environ['MEMCACHE_PASSWORD'] = os.environ['MEMCACHIER_PASSWORD']7 return {8 'default': {9 'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',10 'TIMEOUT': 300,11 'BINARY': True,12 'OPTIONS': { 'tcp_nodelay': True }13 }14 }15 except:16 return {17 'default': {18 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'19 }20 }2122CACHES = get_cache()23CACHE_MIDDLEWARE_ALIAS = 'default'24CACHE_MIDDLEWARE_SECONDS = 30025CACHE_MIDDLEWARE_KEY_PREFIX = ''
Then add the following to MIDDLEWARE_CLASSES:
1 'django.middleware.cache.UpdateCacheMiddleware',2 'django.middleware.common.CommonMiddleware',3 'django.middleware.cache.FetchFromCacheMiddleware',
That's it! The first section configures the application to use Memcached to cache the content when running on Heroku, and sets some configuration parameters, while the second section tells Django to use the per-site cache in order to cache all the site content.
Now, Heroku doesn't include Memcached by default - instead it's available as an add-on called Memcachier. To use add-ons you need to set up a credit card for billing. We will set it up to use the free developer plan, but if you outgrow this you can easily switch to a paid plan. To add Memcachier, run this command:
$ heroku addons:add memcachier:dev
Please note that Memcachier can take a few minutes to get set up, so you may want to leave it a little while between adding it and pushing up your changes. Now we'll commit our changes:
1$ git add requirements.txt django_tutorial_blog_ng/settings.py2$ git commit -m 'Implemented caching with Memcached'
Then we'll push them up to our remote repository and to Heroku:
1$ git push origin master2$ git push heroku master
And that's all you need to do to set up Memcached. In addition to storing your query results in Memcached, enabling the caching framework in Django will also set various HTTP headers to enable web proxies and browsers to cache content for an appropriate length of time. If you open up your browser's developer tools and compare the response headers for your homepage on the latest version of the code with the previous version, you'll notice that a number of additional headers appear, including Cache-Control, Expires and Last-Modified. These tell web browsers and web proxies how often to request the latest version of the HTML document, in order to help you reduce the bandwidth used.
As you can see, for a site like this where you are the only person adding content, it's really easy to implement caching with Django, and for a blog there's very little reason not to do it. If you're not using Heroku and are instead hosting your site on a VPS, then the configuration will be somewhat different - see here for details. You can also find information on using other cache backends on the same page.
That isn't all you can do to speed up your site. Heroku doesn't seem to be very good for serving static files, and if your site is attracting a lot of traffic you might want to host your static files elsewhere, such as on Amazon's S3 service. Doing so is outside the scope of this tutorial, but for that use case, you should check out django-storages.
Clearing the cache automatically
There is one issue with this implementation. As it is right now, if you view the home page, add a post, then reload the page, you may not see the new post immediately because the cache will continue serving the old version until it has expired. That behaviour is less than ideal - we would like the cache to be cleared automatically when a new post gets added so that users will see the new version immediately. That response will still be cached afterwards, so it only means one extra query.
This is the ideal place to introduce signals. Signals are a way to carry out a given action when an event takes place. In our case, we plan to clear the cache when a post is saved (either created or updated).
Note that as we'll be testing the behaviour of the cache at this point, you'll need to install Memcached on your local machine, and we'll need to change the settings to fall back to our local Memcached instance:
1def get_cache():2 import os3 try:4 os.environ['MEMCACHE_SERVERS'] = os.environ['MEMCACHIER_SERVERS'].replace(',', ';')5 os.environ['MEMCACHE_USERNAME'] = os.environ['MEMCACHIER_USERNAME']6 os.environ['MEMCACHE_PASSWORD'] = os.environ['MEMCACHIER_PASSWORD']7 return {8 'default': {9 'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',10 'TIMEOUT': 300,11 'BINARY': True,12 'OPTIONS': { 'tcp_nodelay': True }13 }14 }15 except:16 return {17 'default': {18 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',19 'LOCATION': '127.0.0.1:11211'20 }21 }2223CACHES = get_cache()24CACHE_MIDDLEWARE_ALIAS = 'default'25CACHE_MIDDLEWARE_SECONDS = 30026CACHE_MIDDLEWARE_KEY_PREFIX = ''
If you don't want to install Memcached locally, you can skip this step, but be aware that the test we write for clearing the cache will always pass if you do skip it.
Then we'll run our tests to make sure nothing has been broken:
1$ python manage.py jenkins blogengine2Creating test database for alias 'default'...3.......................4----------------------------------------------------------------------5Ran 23 tests in 6.164s67OK
Let's commit:
1$ git add django_tutorial_blog_ng/settings.py2$ git commit -m 'Now use Memcached in development'
Now we'll add a test for clearing the cache to PostViewTest:
1 def test_clear_cache(self):2 # Create the category3 category = Category()4 category.name = 'python'5 category.description = 'The Python programming language'6 category.save()78 # Create the tag9 tag = Tag()10 tag.name = 'perl'11 tag.description = 'The Perl programming language'12 tag.save()1314 # Create the author15 author = User.objects.create_user('testuser', 'user@example.com', 'password')16 author.save()1718 # Create the site19 site = Site()20 site.name = 'example.com'21 site.domain = 'example.com'22 site.save()2324 # Create the first post25 post = Post()26 post.title = 'My first post'27 post.text = 'This is [my first blog post](http://127.0.0.1:8000/)'28 post.slug = 'my-first-post'29 post.pub_date = timezone.now()30 post.author = author31 post.site = site32 post.category = category33 post.save()34 post.tags.add(tag)3536 # Check new post saved37 all_posts = Post.objects.all()38 self.assertEquals(len(all_posts), 1)3940 # Fetch the index41 response = self.client.get('/')42 self.assertEquals(response.status_code, 200)4344 # Create the second post45 post = Post()46 post.title = 'My second post'47 post.text = 'This is [my second blog post](http://127.0.0.1:8000/)'48 post.slug = 'my-second-post'49 post.pub_date = timezone.now()50 post.author = author51 post.site = site52 post.category = category53 post.save()54 post.tags.add(tag)5556 # Fetch the index again57 response = self.client.get('/')5859 # Check second post present60 self.assertTrue('my second blog post' in response.content)
This should be fairly self-explanatory. We create one post, and request the index page. We then add a second post, request the index page again, and check for the second post. The test should fail because the cached version is returned, rather than the version in the database.
Now we have a test in place, we can implement a fix. First, add this to the top of your models.py:
1from django.db.models.signals import post_save2from django.core.cache import cache
Then add the following at the bottom of the file:
12# Define signals3def new_post(sender, instance, created, **kwargs):4 cache.clear()56# Set up signals7post_save.connect(new_post, sender=Post)
This is fairly straightforward. What we're doing is first defining a function called new_post that is called when a new post is created. We then connect it to the post_save signal. When a post is saved, it calls new_post, which clears the cache, making sure users are seeing the latest and greatest version of your site immediately.
Let's test it:
1$ python manage.py jenkins blogengine2Creating test database for alias 'default'...3........................4----------------------------------------------------------------------5Ran 24 tests in 8.473s67OK8Destroying test database for alias 'default'...
There are a number of signals available, and when you create one, you have access to the created object via the instance parameter. Using signals you can implement all kinds of functionality. For instance, you could implement the functionality to send an email when a new post is published.
If you're using Travis CI, you'll also need to update the config file:
1language: python2python:3- "2.7"4services: memcached5before_install:6 - sudo apt-get install -y libmemcached-dev7# command to install dependencies8install: "pip install -r requirements.txt"9# command to run tests10script: coverage run --include="blogengine/*" --omit="blogengine/migrations/*" manage.py test blogengine11after_success:12 coveralls
Time to commit:
1$ git add blogengine/ .travis.yml2$ git commit -m 'Now clear cache when post added'
Formatting for RSS feeds
Now, we want to offer more than one option for RSS feeds. For instance, if your blog is aggregated on a site such as Planet Python, but you also blog about JavaScript, you may want to be able to provide a feed for posts in the python category only.
If you have written any posts that use any of Markdown's custom formatting, you may notice that if you load your RSS feed in a reader, it isn't formatted as Markdown. Let's fix that. First we'll amend our test:
1class FeedTest(BaseAcceptanceTest):2 def test_all_post_feed(self):3 # Create the category4 category = Category()5 category.name = 'python'6 category.description = 'The Python programming language'7 category.save()89 # Create the tag10 tag = Tag()11 tag.name = 'python'12 tag.description = 'The Python programming language'13 tag.save()1415 # Create the author16 author = User.objects.create_user('testuser', 'user@example.com', 'password')17 author.save()1819 # Create the site20 site = Site()21 site.name = 'example.com'22 site.domain = 'example.com'23 site.save()2425 # Create a post26 post = Post()27 post.title = 'My first post'28 post.text = 'This is my *first* blog post'29 post.slug = 'my-first-post'30 post.pub_date = timezone.now()31 post.author = author32 post.site = site33 post.category = category3435 # Save it36 post.save()3738 # Add the tag39 post.tags.add(tag)40 post.save()4142 # Check we can find it43 all_posts = Post.objects.all()44 self.assertEquals(len(all_posts), 1)45 only_post = all_posts[0]46 self.assertEquals(only_post, post)4748 # Fetch the feed49 response = self.client.get('/feeds/posts/')50 self.assertEquals(response.status_code, 200)5152 # Parse the feed53 feed = feedparser.parse(response.content)5455 # Check length56 self.assertEquals(len(feed.entries), 1)5758 # Check post retrieved is the correct one59 feed_post = feed.entries[0]60 self.assertEquals(feed_post.title, post.title)61 self.assertTrue('This is my <em>first</em> blog post' in feed_post.description)
Don't forget to run the tests to make sure they fail. Now, let's fix it:
1from django.shortcuts import render2from django.views.generic import ListView3from blogengine.models import Category, Post, Tag4from django.contrib.syndication.views import Feed5from django.utils.encoding import force_unicode6from django.utils.safestring import mark_safe7import markdown289# Create your views here.10class CategoryListView(ListView):11 def get_queryset(self):12 slug = self.kwargs['slug']13 try:14 category = Category.objects.get(slug=slug)15 return Post.objects.filter(category=category)16 except Category.DoesNotExist:17 return Post.objects.none()181920class TagListView(ListView):21 def get_queryset(self):22 slug = self.kwargs['slug']23 try:24 tag = Tag.objects.get(slug=slug)25 return tag.post_set.all()26 except Tag.DoesNotExist:27 return Post.objects.none()282930class PostsFeed(Feed):31 title = "RSS feed - posts"32 link = "feeds/posts/"33 description = "RSS feed - blog posts"3435 def items(self):36 return Post.objects.order_by('-pub_date')3738 def item_title(self, item):39 return item.title4041 def item_description(self, item):42 extras = ["fenced-code-blocks"]43 content = mark_safe(markdown2.markdown(force_unicode(item.text),44 extras = extras))45 return content
All we're doing here is amending the item_description method of PostsFeed to render it as Markdown. Now let's run our tests again:
1$ python manage.py jenkins2Creating test database for alias 'default'...3........................4----------------------------------------------------------------------5Ran 24 tests in 9.370s67OK8Destroying test database for alias 'default'...
With that done, we'll commit our changes:
1$ git add blogengine/2$ git commit -m 'Fixed rendering for post feed'
Refactoring our tests
Now, before we get into implementing the feed, our tests are a bit verbose. We create a lot of items over and over again - let's sort that out. Factory Boy is a handy Python module that allows you to create easy-to-use factories for creating objects over and over again in tests. Let's install it:
1$ pip install factory_boy2$ pip freeze > requirements.txt3$ git add requirements.txt4$ git commit -m 'Installed Factory Boy'
Now let's set up a factory for creating posts. Add this at the top of the test file:
import factory.django
Then, before your actual tests, insert the following:
1# Factories2class SiteFactory(factory.django.DjangoModelFactory):3 class Meta:4 model = Site5 django_get_or_create = (6 'name',7 'domain'8 )910 name = 'example.com'11 domain = 'example.com'
Now, wherever you call Site(), add its attributes, and save it, replace those lines with the following:
site = SiteFactory()
Much simpler and more concise, I'm sure you'll agree! Now, let's run the tests to make sure they aren't broken:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3........................4----------------------------------------------------------------------5Ran 24 tests in 7.482s67OK8Destroying test database for alias 'default'...
Let's commit again:
1$ git add blogengine/tests.py2$ git commit -m 'Now use Factory Boy for site objects'
Let's do the same thing with Category objects:
1class CategoryFactory(factory.django.DjangoModelFactory):2 class Meta:3 model = Category4 django_get_or_create = (5 'name',6 'description',7 'slug'8 )910 name = 'python'11 description = 'The Python programming language'12 slug = 'python'
Again, just find every time we call Category() and replace it with the following:
category = CategoryFactory()
Now if we run our tests, we'll notice a serious error:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3EE..EE.EE.EE...E.EEE..E.4======================================================================5ERROR: test_create_category (blogengine.tests.PostTest)6----------------------------------------------------------------------7Traceback (most recent call last):8 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 42, in test_create_category9 category = CategoryFactory()10 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__11 return cls.create(**kwargs)12 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create13 return cls._generate(True, attrs)14 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate15 obj = cls._prepare(create, **attrs)16 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare17 return cls._create(model_class, *args, **kwargs)18 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create19 return cls._get_or_create(model_class, *args, **kwargs)20 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create21 obj, _created = manager.get_or_create(*args, **key_fields)22 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create23 return self.get_queryset().get_or_create(**kwargs)24 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create25 obj.save(force_insert=True, using=self.db)26TypeError: save() got an unexpected keyword argument 'force_insert'2728======================================================================29ERROR: test_create_post (blogengine.tests.PostTest)30----------------------------------------------------------------------31Traceback (most recent call last):32 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 80, in test_create_post33 category = CategoryFactory()34 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__35 return cls.create(**kwargs)36 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create37 return cls._generate(True, attrs)38 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate39 obj = cls._prepare(create, **attrs)40 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare41 return cls._create(model_class, *args, **kwargs)42 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create43 return cls._get_or_create(model_class, *args, **kwargs)44 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create45 obj, _created = manager.get_or_create(*args, **key_fields)46 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create47 return self.get_queryset().get_or_create(**kwargs)48 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create49 obj.save(force_insert=True, using=self.db)50TypeError: save() got an unexpected keyword argument 'force_insert'5152======================================================================53ERROR: test_create_post (blogengine.tests.AdminTest)54----------------------------------------------------------------------55Traceback (most recent call last):56 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 339, in test_create_post57 category = CategoryFactory()58 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__59 return cls.create(**kwargs)60 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create61 return cls._generate(True, attrs)62 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate63 obj = cls._prepare(create, **attrs)64 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare65 return cls._create(model_class, *args, **kwargs)66 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create67 return cls._get_or_create(model_class, *args, **kwargs)68 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create69 obj, _created = manager.get_or_create(*args, **key_fields)70 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create71 return self.get_queryset().get_or_create(**kwargs)72 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create73 obj.save(force_insert=True, using=self.db)74TypeError: save() got an unexpected keyword argument 'force_insert'7576======================================================================77ERROR: test_create_post_without_tag (blogengine.tests.AdminTest)78----------------------------------------------------------------------79Traceback (most recent call last):80 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 378, in test_create_post_without_tag81 category = CategoryFactory()82 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__83 return cls.create(**kwargs)84 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create85 return cls._generate(True, attrs)86 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate87 obj = cls._prepare(create, **attrs)88 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare89 return cls._create(model_class, *args, **kwargs)90 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create91 return cls._get_or_create(model_class, *args, **kwargs)92 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create93 obj, _created = manager.get_or_create(*args, **key_fields)94 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create95 return self.get_queryset().get_or_create(**kwargs)96 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create97 obj.save(force_insert=True, using=self.db)98TypeError: save() got an unexpected keyword argument 'force_insert'99100======================================================================101ERROR: test_delete_category (blogengine.tests.AdminTest)102----------------------------------------------------------------------103Traceback (most recent call last):104 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 245, in test_delete_category105 category = CategoryFactory()106 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__107 return cls.create(**kwargs)108 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create109 return cls._generate(True, attrs)110 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate111 obj = cls._prepare(create, **attrs)112 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare113 return cls._create(model_class, *args, **kwargs)114 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create115 return cls._get_or_create(model_class, *args, **kwargs)116 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create117 obj, _created = manager.get_or_create(*args, **key_fields)118 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create119 return self.get_queryset().get_or_create(**kwargs)120 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create121 obj.save(force_insert=True, using=self.db)122TypeError: save() got an unexpected keyword argument 'force_insert'123124======================================================================125ERROR: test_delete_post (blogengine.tests.AdminTest)126----------------------------------------------------------------------127Traceback (most recent call last):128 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 467, in test_delete_post129 category = CategoryFactory()130 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__131 return cls.create(**kwargs)132 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create133 return cls._generate(True, attrs)134 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate135 obj = cls._prepare(create, **attrs)136 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare137 return cls._create(model_class, *args, **kwargs)138 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create139 return cls._get_or_create(model_class, *args, **kwargs)140 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create141 obj, _created = manager.get_or_create(*args, **key_fields)142 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create143 return self.get_queryset().get_or_create(**kwargs)144 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create145 obj.save(force_insert=True, using=self.db)146TypeError: save() got an unexpected keyword argument 'force_insert'147148======================================================================149ERROR: test_edit_category (blogengine.tests.AdminTest)150----------------------------------------------------------------------151Traceback (most recent call last):152 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 221, in test_edit_category153 category = CategoryFactory()154 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__155 return cls.create(**kwargs)156 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create157 return cls._generate(True, attrs)158 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate159 obj = cls._prepare(create, **attrs)160 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare161 return cls._create(model_class, *args, **kwargs)162 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create163 return cls._get_or_create(model_class, *args, **kwargs)164 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create165 obj, _created = manager.get_or_create(*args, **key_fields)166 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create167 return self.get_queryset().get_or_create(**kwargs)168 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create169 obj.save(force_insert=True, using=self.db)170TypeError: save() got an unexpected keyword argument 'force_insert'171172======================================================================173ERROR: test_edit_post (blogengine.tests.AdminTest)174----------------------------------------------------------------------175Traceback (most recent call last):176 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 410, in test_edit_post177 category = CategoryFactory()178 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__179 return cls.create(**kwargs)180 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create181 return cls._generate(True, attrs)182 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate183 obj = cls._prepare(create, **attrs)184 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare185 return cls._create(model_class, *args, **kwargs)186 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create187 return cls._get_or_create(model_class, *args, **kwargs)188 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create189 obj, _created = manager.get_or_create(*args, **key_fields)190 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create191 return self.get_queryset().get_or_create(**kwargs)192 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create193 obj.save(force_insert=True, using=self.db)194TypeError: save() got an unexpected keyword argument 'force_insert'195196======================================================================197ERROR: test_all_post_feed (blogengine.tests.FeedTest)198----------------------------------------------------------------------199Traceback (most recent call last):200 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 810, in test_all_post_feed201 category = CategoryFactory()202 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__203 return cls.create(**kwargs)204 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create205 return cls._generate(True, attrs)206 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate207 obj = cls._prepare(create, **attrs)208 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare209 return cls._create(model_class, *args, **kwargs)210 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create211 return cls._get_or_create(model_class, *args, **kwargs)212 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create213 obj, _created = manager.get_or_create(*args, **key_fields)214 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create215 return self.get_queryset().get_or_create(**kwargs)216 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create217 obj.save(force_insert=True, using=self.db)218TypeError: save() got an unexpected keyword argument 'force_insert'219220======================================================================221ERROR: test_category_page (blogengine.tests.PostViewTest)222----------------------------------------------------------------------223Traceback (most recent call last):224 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 640, in test_category_page225 category = CategoryFactory()226 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__227 return cls.create(**kwargs)228 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create229 return cls._generate(True, attrs)230 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate231 obj = cls._prepare(create, **attrs)232 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare233 return cls._create(model_class, *args, **kwargs)234 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create235 return cls._get_or_create(model_class, *args, **kwargs)236 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create237 obj, _created = manager.get_or_create(*args, **key_fields)238 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create239 return self.get_queryset().get_or_create(**kwargs)240 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create241 obj.save(force_insert=True, using=self.db)242TypeError: save() got an unexpected keyword argument 'force_insert'243244======================================================================245ERROR: test_clear_cache (blogengine.tests.PostViewTest)246----------------------------------------------------------------------247Traceback (most recent call last):248 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 753, in test_clear_cache249 category = CategoryFactory()250 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__251 return cls.create(**kwargs)252 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create253 return cls._generate(True, attrs)254 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate255 obj = cls._prepare(create, **attrs)256 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare257 return cls._create(model_class, *args, **kwargs)258 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create259 return cls._get_or_create(model_class, *args, **kwargs)260 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create261 obj, _created = manager.get_or_create(*args, **key_fields)262 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create263 return self.get_queryset().get_or_create(**kwargs)264 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create265 obj.save(force_insert=True, using=self.db)266TypeError: save() got an unexpected keyword argument 'force_insert'267268======================================================================269ERROR: test_index (blogengine.tests.PostViewTest)270----------------------------------------------------------------------271Traceback (most recent call last):272 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 518, in test_index273 category = CategoryFactory()274 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__275 return cls.create(**kwargs)276 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create277 return cls._generate(True, attrs)278 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate279 obj = cls._prepare(create, **attrs)280 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare281 return cls._create(model_class, *args, **kwargs)282 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create283 return cls._get_or_create(model_class, *args, **kwargs)284 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create285286287 obj, _created = manager.get_or_create(*args, **key_fields)288 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create289 return self.get_queryset().get_or_create(**kwargs)290 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create291 obj.save(force_insert=True, using=self.db)292TypeError: save() got an unexpected keyword argument 'force_insert'293294======================================================================295ERROR: test_post_page (blogengine.tests.PostViewTest)296----------------------------------------------------------------------297Traceback (most recent call last):298 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 576, in test_post_page299 category = CategoryFactory()300 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 82, in __call__301 return cls.create(**kwargs)302 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 585, in create303 return cls._generate(True, attrs)304 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 510, in _generate305 obj = cls._prepare(create, **attrs)306 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/base.py", line 485, in _prepare307 return cls._create(model_class, *args, **kwargs)308 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 151, in _create309310311 return cls._get_or_create(model_class, *args, **kwargs)312 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/factory/django.py", line 142, in _get_or_create313 obj, _created = manager.get_or_create(*args, **key_fields)314 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 154, in get_or_create315 return self.get_queryset().get_or_create(**kwargs)316 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/venv/lib/python2.7/site-packages/django/db/models/query.py", line 383, in get_or_create317 obj.save(force_insert=True, using=self.db)318TypeError: save() got an unexpected keyword argument 'force_insert'319320----------------------------------------------------------------------321Ran 24 tests in 5.162s322323FAILED (errors=13)324Destroying test database for alias 'default'...
Thankfully, this is easy to fix. We just need to amend the custom save() method of the Category model:
1 def save(self, *args, **kwargs):2 if not self.slug:3 self.slug = slugify(unicode(self.name))4 super(Category, self).save(*args, **kwargs)
That should resolve the issue:
1$ python manage.py jenkins2Creating test database for alias 'default'...3........................4----------------------------------------------------------------------5Ran 24 tests in 7.749s67OK8Destroying test database for alias 'default'...
Let's commit again:
1$ git add blogengine/2$ git commit -m 'Category now uses Factory Boy'
Now let's do the same thing for tags:
1class TagFactory(factory.django.DjangoModelFactory):2 class Meta:3 model = Tag4 django_get_or_create = (5 'name',6 'description',7 'slug'8 )910 name = 'python'11 description = 'The Python programming language'12 slug = 'python'
And replace the sections where we create new Tag objects:
tag = TagFactory()
Note that some tags have different values. We can easily pass different values to our TagFactory() to override the default values:
tag = TagFactory(name='perl', description='The Perl programming language')
The Tag model has the same issue as the Category one did, so let's fix that:
1 def save(self, *args, **kwargs):2 if not self.slug:3 self.slug = slugify(unicode(self.name))4 super(Tag, self).save(*args, **kwargs)
We run our tests again:
1$ python manage.py test blogengine/2Creating test database for alias 'default'...3........................4----------------------------------------------------------------------5Ran 24 tests in 7.153s67OK8Destroying test database for alias 'default'...
Time to commit again:
1$ git add blogengine/2$ git commit -m 'Now use Factory Boy for tags'
Next we'll create a factory for adding users. Note that the factory name doesn't have to match the object name, so you can create factories for different types of users. Here we create a factory for authors - you could, for instance, create a separate factory for subscribers if you wanted:
1class AuthorFactory(factory.django.DjangoModelFactory):2 class Meta:3 model = User4 django_get_or_create = ('username','email', 'password',)56 username = 'testuser'7 email = 'user@example.com'8 password = 'password'
And as before, replace those sections where we create users with the following:
author = AuthorFactory()
Run the tests again:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3........................4----------------------------------------------------------------------5Ran 24 tests in 5.808s67OK8Destroying test database for alias 'default'...
We commit our changes:
1$ git add blogengine/2$ git commit -m 'Now use Factory Boy for creating authors'
Now we'll create a flat page factory:
123class FlatPageFactory(factory.django.DjangoModelFactory):4 class Meta:5 model = FlatPage6 django_get_or_create = (7 'url',8 'title',9 'content'10 )1112 url = '/about/'13 title = 'About me'14 content = 'All about me'
And use it for our flat page test:
page = FlatPageFactory()
Check the tests pass:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3........................4----------------------------------------------------------------------5Ran 24 tests in 5.796s67OK8Destroying test database for alias 'default'...
And commit again:
1$ git add blogengine/2$ git commit -m 'Now use Factory Boy for flat page test'
Now we'll create a final factory for posts:
1class PostFactory(factory.django.DjangoModelFactory):2 class Meta:3 model = Post4 django_get_or_create = (5 'title',6 'text',7 'slug',8 'pub_date'9 )1011 title = 'My first post'12 text = 'This is my first blog post'13 slug = 'my-first-post'14 pub_date = timezone.now()15 author = factory.SubFactory(AuthorFactory)16 site = factory.SubFactory(SiteFactory)17 category = factory.SubFactory(CategoryFactory)
This factory is a little bit different. Because our Post model depends on several others, we need to be able to create those additional objects on demand. By designating them as subfactories, we can easily create the associated objects for our Post object.
That means that not only can we get rid of our Post() calls, but we can also get rid of the factory calls to create the associated objects for Post models. Again, I'll leave actually doing this as an exercise for the reader, but you can always refer to the GitHub repository if you're not too sure.
Make sure your tests still pass, then commit the changes:
1$ git add blogengine/2$ git commit -m 'Now use Factory Boy for testing posts'
Using Factory Boy made a big difference to the size of the test file - I was able to cut it down by over 200 lines of code. As your application gets bigger, it gets harder to maintain, so do what you can to keep the size down.
Additional RSS feeds
Now, let's implement our additional RSS feeds. First, we'll write a test for the category feed. Add this to the FeedTest class:
1 def test_category_feed(self):2 # Create a post3 post = PostFactory(text='This is my *first* blog post')45 # Create another post in a different category6 category = CategoryFactory(name='perl', description='The Perl programming language', slug='perl')7 post2 = PostFactory(text='This is my *second* blog post', title='My second post', slug='my-second-post', category=category)89 # Fetch the feed10 response = self.client.get('/feeds/posts/category/python/')11 self.assertEquals(response.status_code, 200)1213 # Parse the feed14 feed = feedparser.parse(response.content)1516 # Check length17 self.assertEquals(len(feed.entries), 1)1819 # Check post retrieved is the correct one20 feed_post = feed.entries[0]21 self.assertEquals(feed_post.title, post.title)22 self.assertTrue('This is my <em>first</em> blog post' in feed_post.description)2324 # Check other post is not in this feed25 self.assertTrue('This is my <em>second</em> blog post' not in response.content)
Here we create two posts in different categories (note that we create a new category and override the post category for it). We then fetch /feeds/posts/category/python/ and assert that it contains only one post, with the content of the first post and not the content of the second.
Run the tests and they should fail:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3................F........4======================================================================5FAIL: test_category_feed (blogengine.tests.FeedTest)6----------------------------------------------------------------------7Traceback (most recent call last):8 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 716, in test_category_feed9 self.assertEquals(response.status_code, 200)10AssertionError: 404 != 2001112----------------------------------------------------------------------13Ran 25 tests in 5.804s1415FAILED (failures=1)16Destroying test database for alias 'default'...
Because we haven't yet implemented that route, we get a 404 error. So let's create a route for this:
1from django.conf.urls import patterns, url2from django.views.generic import ListView, DetailView3from blogengine.models import Post, Category, Tag4from blogengine.views import CategoryListView, TagListView, PostsFeed, CategoryPostsFeed56urlpatterns = patterns('',7 # Index8 url(r'^(?P<page>\d+)?/?$', ListView.as_view(9 model=Post,10 paginate_by=5,11 )),1213 # Individual posts14 url(r'^(?P<pub_date__year>\d{4})/(?P<pub_date__month>\d{1,2})/(?P<slug>[a-zA-Z0-9-]+)/?$', DetailView.as_view(15 model=Post,16 )),1718 # Categories19 url(r'^category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryListView.as_view(20 paginate_by=5,21 model=Category,22 )),2324 # Tags25 url(r'^tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagListView.as_view(26 paginate_by=5,27 model=Tag,28 )),2930 # Post RSS feed31 url(r'^feeds/posts/$', PostsFeed()),3233 # Category RSS feed34 url(r'^feeds/posts/category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryPostsFeed()),35)
Note that the category RSS feed route is similar to the post RSS feed route, but accepts a slug parameter. We will use this to pass through the slug for the category in question. Also note we import the CategoryPostsFeed view. Now, we need to create that view. Fortunately, because it's written as a Python class, we can extend the existing PostsFeed class. Open up your views file and amend it to look like this:
1from django.shortcuts import get_object_or_4042from django.views.generic import ListView3from blogengine.models import Category, Post, Tag4from django.contrib.syndication.views import Feed5from django.utils.encoding import force_unicode6from django.utils.safestring import mark_safe7import markdown289# Create your views here.10class CategoryListView(ListView):11 def get_queryset(self):12 slug = self.kwargs['slug']13 try:14 category = Category.objects.get(slug=slug)15 return Post.objects.filter(category=category)16 except Category.DoesNotExist:17 return Post.objects.none()181920class TagListView(ListView):21 def get_queryset(self):22 slug = self.kwargs['slug']23 try:24 tag = Tag.objects.get(slug=slug)25 return tag.post_set.all()26 except Tag.DoesNotExist:27 return Post.objects.none()282930class PostsFeed(Feed):31 title = "RSS feed - posts"32 description = "RSS feed - blog posts"33 link = '/'3435 def items(self):36 return Post.objects.order_by('-pub_date')3738 def item_title(self, item):39 return item.title4041 def item_description(self, item):42 extras = ["fenced-code-blocks"]43 content = mark_safe(markdown2.markdown(force_unicode(item.text),44 extras = extras))45 return content464748class CategoryPostsFeed(PostsFeed):49 def get_object(self, request, slug):50 return get_object_or_404(Category, slug=slug)5152 def title(self, obj):53 return "RSS feed - blog posts in category %s" % obj.name5455 def link(self, obj):56 return obj.get_absolute_url()5758 def description(self, obj):59 return "RSS feed - blog posts in category %s" % obj.name6061 def items(self, obj):62 return Post.objects.filter(category=obj).order_by('-pub_date')
Note that many of our fields don't have to be explicitly defined as they are inherited from PostsFeed. We can't hard-code the title, link or description because they depend on the category, so we instead define methods to return the appropriate text.
Also note get_object() - we define this so that we can ensure the category exists. If it doesn't exist, then it returns a 404 error rather than showing an empty feed.
We also override items() to filter it to just those posts that are in the given category.
If you run the tests again, they should now pass:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3.........................4----------------------------------------------------------------------5Ran 25 tests in 5.867s67OK8Destroying test database for alias 'default'...
Let's commit our changes:
1$ git add blogengine/2$ git commit -m 'Implemented category RSS feed'
Now, we can get our category RSS feed, but how do we navigate to it? Let's add a link to each category page that directs a user to its RSS feed. To do so, we'll need to create a new template for category pages. First, let's add some code to our tests to ensure that the right template is used at all times. Add the following to the end of the test_index method of PostViewTest:
1 # Check the correct template was used2 self.assertTemplateUsed(response, 'blogengine/post_list.html')
Then, add this to the end of test_post_page:
12 # Check the correct template was used3 self.assertTemplateUsed(response, 'blogengine/post_detail.html')
Finally, add this to the end of test_category_page:
1 # Check the correct template was used2 self.assertTemplateUsed(response, 'blogengine/category_post_list.html')
These assertions confirm which template was used to generate which request.
Next, we head into our views file:
1class CategoryListView(ListView):2 template_name = 'blogengine/category_post_list.html'34 def get_queryset(self):5 slug = self.kwargs['slug']6 try:7 category = Category.objects.get(slug=slug)8 return Post.objects.filter(category=category)9 except Category.DoesNotExist:10 return Post.objects.none()1112 def get_context_data(self, **kwargs):13 context = super(CategoryListView, self).get_context_data(**kwargs)14 slug = self.kwargs['slug']15 try:16 context['category'] = Category.objects.get(slug=slug)17 except Category.DoesNotExist:18 context['category'] = None19 return context
Note that we first of all change the template used by this view. Then, we override get_context_data to add in additional data. What we're doing is getting the slug that was passed through, looking up any category for which it is the slug, and returning it as additional context data. Using this method, you can easily add additional data that you may wish to render in your Django templates.
Finally, we create our new template:
1{% extends "blogengine/includes/base.html" %}23 {% load custom_markdown %}45 {% block content %}6 {% if object_list %}7 {% for post in object_list %}8 <div class="post col-md-12">9 <h1><a href="{{ post.get_absolute_url }}">{{ post.title }}</a></h1>10 <h3>{{ post.pub_date }}</h3>11 {{ post.text|custom_markdown }}12 </div>13 {% if post.category %}14 <div class="col-md-12">15 <a href="{{ post.category.get_absolute_url }}"><span class="label label-primary">{{ post.category.name }}</span></a>16 </div>17 {% endif %}18 {% if post.tags %}19 <div class="col-md-12">20 {% for tag in post.tags.all %}21 <a href="{{ tag.get_absolute_url }}"><span class="label label-success">{{ tag.name }}</span></a>22 {% endfor %}23 </div>24 {% endif %}25 {% endfor %}26 {% else %}27 <p>No posts found</p>28 {% endif %}2930 <ul class="pager">31 {% if page_obj.has_previous %}32 <li class="previous"><a href="/{{ page_obj.previous_page_number }}/">Previous Page</a></li>33 {% endif %}34 {% if page_obj.has_next %}35 <li class="next"><a href="/{{ page_obj.next_page_number }}/">Next Page</a></li>36 {% endif %}37 </ul>3839 <a href="/feeds/posts/category/{{ category.slug }}/">RSS feed for category {{ category.name }}</a>4041 {% endblock %}
Note that the category has been passed through to the template and is now accessible. If you run the tests, they should now pass:
1$ python manage.py jenkins2Creating test database for alias 'default'...3.........................4----------------------------------------------------------------------5Ran 25 tests in 7.232s67OK8Destroying test database for alias 'default'...
With that done. we can commit our changes:
1$ git add templates/ blogengine/2$ git commit -m 'Added link to RSS feed from category page'
Next up, let's implement another RSS feed for tags. First, we'll implement our test:
1 def test_tag_feed(self):2 # Create a post3 post = PostFactory(text='This is my *first* blog post')4 tag = TagFactory()5 post.tags.add(tag)6 post.save()78 # Create another post with a different tag9 tag2 = TagFactory(name='perl', description='The Perl programming language', slug='perl')10 post2 = PostFactory(text='This is my *second* blog post', title='My second post', slug='my-second-post')11 post2.tags.add(tag2)12 post2.save()1314 # Fetch the feed15 response = self.client.get('/feeds/posts/tag/python/')16 self.assertEquals(response.status_code, 200)1718 # Parse the feed19 feed = feedparser.parse(response.content)2021 # Check length22 self.assertEquals(len(feed.entries), 1)2324 # Check post retrieved is the correct one25 feed_post = feed.entries[0]26 self.assertEquals(feed_post.title, post.title)27 self.assertTrue('This is my <em>first</em> blog post' in feed_post.description)2829 # Check other post is not in this feed30 self.assertTrue('This is my <em>second</em> blog post' not in response.content)
This is virtually identical to the test for the category feed, but we adjust it to work with the Tag attribute and change the URL. Let's check that our test fails:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3.................F........4======================================================================5FAIL: test_tag_feed (blogengine.tests.FeedTest)6----------------------------------------------------------------------7Traceback (most recent call last):8 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 757, in test_tag_feed9 self.assertEquals(response.status_code, 200)10AssertionError: 404 != 2001112----------------------------------------------------------------------13Ran 26 tests in 5.760s1415FAILED (failures=1)16Destroying test database for alias 'default'...
As before, we create a route for this:
1from django.conf.urls import patterns, url2from django.views.generic import ListView, DetailView3from blogengine.models import Post, Category, Tag4from blogengine.views import CategoryListView, TagListView, PostsFeed, CategoryPostsFeed, TagPostsFeed56urlpatterns = patterns('',7 # Index8 url(r'^(?P<page>\d+)?/?$', ListView.as_view(9 model=Post,10 paginate_by=5,11 )),1213 # Individual posts14 url(r'^(?P<pub_date__year>\d{4})/(?P<pub_date__month>\d{1,2})/(?P<slug>[a-zA-Z0-9-]+)/?$', DetailView.as_view(15 model=Post,16 )),1718 # Categories19 url(r'^category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryListView.as_view(20 paginate_by=5,21 model=Category,22 )),2324 # Tags25 url(r'^tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagListView.as_view(26 paginate_by=5,27 model=Tag,28 )),2930 # Post RSS feed31 url(r'^feeds/posts/$', PostsFeed()),3233 # Category RSS feed34 url(r'^feeds/posts/category/(?P<slug>[a-zA-Z0-9-]+)/?$', CategoryPostsFeed()),3536 # Tag RSS feed37 url(r'^feeds/posts/tag/(?P<slug>[a-zA-Z0-9-]+)/?$', TagPostsFeed()),38)
Next, we create our view:
1class TagPostsFeed(PostsFeed):2 def get_object(self, request, slug):3 return get_object_or_404(Tag, slug=slug)45 def title(self, obj):6 return "RSS feed - blog posts tagged %s" % obj.name78 def link(self, obj):9 return obj.get_absolute_url()1011 def description(self, obj):12 return "RSS feed - blog posts tagged %s" % obj.name1314 def items(self, obj):15 try:16 tag = Tag.objects.get(slug=obj.slug)17 return tag.post_set.all()18 except Tag.DoesNotExist:19 return Post.objects.none()
Again, this inherits from PostsFeed, but the syntax for getting posts matching a tag is slightly different because they use a many-to-many relationship.
We also need a template for the tag pages. Add this to the end of the test_tag_page method:
12 # Check the correct template was used3 self.assertTemplateUsed(response, 'blogengine/tag_post_list.html')
Let's create that template:
1{% extends "blogengine/includes/base.html" %}23 {% load custom_markdown %}45 {% block content %}6 {% if object_list %}7 {% for post in object_list %}8 <div class="post col-md-12">9 <h1><a href="{{ post.get_absolute_url }}">{{ post.title }}</a></h1>10 <h3>{{ post.pub_date }}</h3>11 {{ post.text|custom_markdown }}12 </div>13 {% if post.category %}14 <div class="col-md-12">15 <a href="{{ post.category.get_absolute_url }}"><span class="label label-primary">{{ post.category.name }}</span></a>16 </div>17 {% endif %}18 {% if post.tags %}19 <div class="col-md-12">20 {% for tag in post.tags.all %}21 <a href="{{ tag.get_absolute_url }}"><span class="label label-success">{{ tag.name }}</span></a>22 {% endfor %}23 </div>24 {% endif %}25 {% endfor %}26 {% else %}27 <p>No posts found</p>28 {% endif %}2930 <ul class="pager">31 {% if page_obj.has_previous %}32 <li class="previous"><a href="/{{ page_obj.previous_page_number }}/">Previous Page</a></li>33 {% endif %}34 {% if page_obj.has_next %}35 <li class="next"><a href="/{{ page_obj.next_page_number }}/">Next Page</a></li>36 {% endif %}37 </ul>3839 <a href="/feeds/posts/tag/{{ tag.slug }}/">RSS feed for tag {{ tag.name }}</a>4041 {% endblock %}
This is virtually identical to the category template. You'll also need to apply this template in the view for the tag list, and pass the tag name through as context data:
1class TagListView(ListView):2 template_name = 'blogengine/tag_post_list.html'34 def get_queryset(self):5 slug = self.kwargs['slug']6 try:7 tag = Tag.objects.get(slug=slug)8 return tag.post_set.all()9 except Tag.DoesNotExist:10 return Post.objects.none()1112 def get_context_data(self, **kwargs):13 context = super(TagListView, self).get_context_data(**kwargs)14 slug = self.kwargs['slug']15 try:16 context['tag'] = Tag.objects.get(slug=slug)17 except Tag.DoesNotExist:18 context['tag'] = None19 return context
Let's run our tests:
1$ python manage.py test blogengine2Creating test database for alias 'default'...3..........................4----------------------------------------------------------------------5Ran 26 tests in 5.770s67OK8Destroying test database for alias 'default'...
You may want to do a quick check to ensure your tag feed link works as expected. Time to commit:
1$ git add blogengine templates2$ git commit -m 'Implemented tag feeds'
Moving our templates
Before we crack on with implementing search, there's one more piece of housekeeping. In Django, templates can be applied at project level or at app level. So far, we've been storing them in the project, but we would like our app to be as self-contained as possible so it can just be dropped into future projects where we need a blog. That way, it can be easily overridden for specific projects. You can move the folders and update the Git repository at the same time with this command:
$ git mv templates/ blogengine/
We run the tests to make sure nothing untoward has happened:
1$ python manage.py test2Creating test database for alias 'default'...3..........................4----------------------------------------------------------------------5Ran 26 tests in 5.847s67OK8Destroying test database for alias 'default'...
And we commit:
$ git commit -m 'Moved templates'
Note that git mv updates Git and moves the files, so you don't need to call git add.
Implementing search
For our final task today, we will be implementing a very simple search engine. Our requirements are:
- It should be in the header, to allow for easy access from anywhere in the front end.
- It should search the title and text of posts.
First, we'll write our tests:
1class SearchViewTest(BaseAcceptanceTest):2 def test_search(self):3 # Create a post4 post = PostFactory()56 # Create another post7 post2 = PostFactory(text='This is my *second* blog post', title='My second post', slug='my-second-post')89 # Search for first post10 response = self.client.get('/search?q=first')11 self.assertEquals(response.status_code, 200)1213 # Check the first post is contained in the results14 self.assertTrue('My first post' in response.content)1516 # Check the second post is not contained in the results17 self.assertTrue('My second post' not in response.content)1819 # Search for second post20 response = self.client.get('/search?q=second')21 self.assertEquals(response.status_code, 200)2223 # Check the first post is not contained in the results24 self.assertTrue('My first post' not in response.content)2526 # Check the second post is contained in the results27 self.assertTrue('My second post' in response.content)
Don't forget to run the tests to make sure they fail:
1$ python manage.py test2Creating test database for alias 'default'...3..........................F4======================================================================5FAIL: test_search (blogengine.tests.SearchViewTest)6----------------------------------------------------------------------7Traceback (most recent call last):8 File "/Users/matthewdaly/Projects/django_tutorial_blog_ng/blogengine/tests.py", line 819, in test_search9 self.assertEquals(response.status_code, 200)10AssertionError: 404 != 2001112----------------------------------------------------------------------13Ran 27 tests in 6.919s1415FAILED (failures=1)16Destroying test database for alias 'default'...
With that done, we can add the search form to the header:
1<!DOCTYPE html>2<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->3<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->4<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->5<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->6 <head>7 <meta charset="utf-8">8 <meta http-equiv="X-UA-Compatible" content="IE=edge">9 <title>{% block title %}My Django Blog{% endblock %}</title>10 <meta name="description" content="">11 <meta name="viewport" content="width=device-width, initial-scale=1">12 <link rel="alternate" type="application/rss+xml" title="Blog posts" href="/feeds/posts/" >1314 <!-- Place favicon.ico and apple-touch-icon.png in the root directory -->1516 {% load staticfiles %}17 <link rel="stylesheet" href="{% static 'bower_components/html5-boilerplate/css/normalize.css' %}">18 <link rel="stylesheet" href="{% static 'bower_components/html5-boilerplate/css/main.css' %}">19 <link rel="stylesheet" href="{% static 'bower_components/bootstrap/dist/css/bootstrap.min.css' %}">20 <link rel="stylesheet" href="{% static 'bower_components/bootstrap/dist/css/bootstrap-theme.min.css' %}">21 <link rel="stylesheet" href="{% static 'css/main.css' %}">22 <link rel="stylesheet" href="{% static 'css/code.css' %}">23 <script src="{% static 'bower_components/html5-boilerplate/js/vendor/modernizr-2.6.2.min.js' %}"></script>24 </head>25 <body>26 <!--[if lt IE 7]>27 <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>28 <![endif]-->2930 <!-- Add your site or application content here -->3132 <div id="fb-root"></div>33 <script>(function(d, s, id) {34 var js, fjs = d.getElementsByTagName(s)[0];35 if (d.getElementById(id)) return;36 js = d.createElement(s); js.id = id;37 js.src = "//connect.facebook.net/en_GB/all.js#xfbml=1";38 fjs.parentNode.insertBefore(js, fjs);39 }(document, 'script', 'facebook-jssdk'));</script>4041 <div class="navbar navbar-static-top navbar-inverse">42 <div class="container-fluid">43 <div class="navbar-header">44 <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#header-nav">45 <span class="icon-bar"></span>46 <span class="icon-bar"></span>47 <span class="icon-bar"></span>48 </button>49 <a class="navbar-brand" href="/">My Django Blog</a>50 </div>51 <div class="collapse navbar-collapse" id="header-nav">52 <ul class="nav navbar-nav">53 {% load flatpages %}54 {% get_flatpages as flatpages %}55 {% for flatpage in flatpages %}56 <li><a href="{{ flatpage.url }}">{{ flatpage.title }}</a></li>57 {% endfor %}58 <li><a href="/feeds/posts/">RSS feed</a></li>5960 <form action="/search" method="GET" class="navbar-form navbar-left">61 <div class="form-group">62 <input type="text" name="q" placeholder="Search..." class="form-control"></input>63 </div>64 <button type="submit" class="btn btn-default">Search</button>65 </form>66 </ul>67 </div>68 </div>69 </div>7071 <div class="container">72 {% block header %}73 <div class="page-header">74 <h1>My Django Blog</h1>75 </div>76 {% endblock %}7778 <div class="row">79 {% block content %}{% endblock %}80 </div>81 </div>8283 <div class="container footer">84 <div class="row">85 <div class="span12">86 <p>Copyright © {% now "Y" %}</p>87 </div>88 </div>89 </div>9091 <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>92 <script>window.jQuery || document.write('<script src="{% static 'bower_components/html5-boilerplate/js/vendor/jquery-1.10.2.min.js' %}"><\/script>')</script>93 <script src="{% static 'bower_components/html5-boilerplate/js/plugins.js' %}"></script>94 <script src="{% static 'bower_components/bootstrap/dist/js/bootstrap.min.js' %}"></script>9596 <!-- Google Analytics: change UA-XXXXX-X to be your site's ID. -->97 <script>98 (function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=99 function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;100 e=o.createElement(i);r=o.getElementsByTagName(i)[0];101 e.src='//www.google-analytics.com/analytics.js';102 r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));103 ga('create','UA-XXXXX-X');ga('send','pageview');104 </script>105 </body>106</html>
Now we'll actually implement our search. Implementing search using Django's generic views can be fiddly, so we'll write our search view as a function instead. First, amend the imports at the top of your view file to look like this:
1from django.shortcuts import get_object_or_404, render_to_response2from django.core.paginator import Paginator, EmptyPage3from django.db.models import Q4from django.views.generic import ListView5from blogengine.models import Category, Post, Tag6from django.contrib.syndication.views import Feed7from django.utils.encoding import force_unicode8from django.utils.safestring import mark_safe9import markdown2
Next, add the following code to the end of the file:
1def getSearchResults(request):2 """3 Search for a post by title or text4 """5 # Get the query data6 query = request.GET.get('q', '')7 page = request.GET.get('page', 1)89 # Query the database10 if query:11 results = Post.objects.filter(Q(text__icontains=query) | Q(title__icontains=query))12 else:13 results = None1415 # Add pagination16 pages = Paginator(results, 5)1718 # Get specified page19 try:20 returned_page = pages.page(page)21 except EmptyPage:22 returned_page = pages.page(pages.num_pages)2324 # Display the search results25 return render_to_response('blogengine/search_post_list.html',26 {'page_obj': returned_page,27 'object_list': returned_page.object_list,28 'search': query})
As this is the first time we've written a view without using generic views, a little explanation is called for. First we get the values of the q and page parameters passed to the view. q contains the query text and page contains the page number. Note also that our page defaults to 1 if not set.
We then use the Q object to perform a query. The Django ORM will AND together keyword argument queries, but that's not the behaviour we want here. Instead we want to be able to search for content in the title or text, so we need to use a query with an OR statement, which necessitates using the Q object.
Next, we use the Paginator object to manually paginate the results, and if it raises an EmptyPage exception, to just show the last page instead. Finally we render the template blogengine/search_post_list.html, and pass through the parameters page_obj for the returned page, object_list for the objects, and search for the query.
We also need to add a route for our new view:
12 # Search posts3 url(r'^search', 'blogengine.views.getSearchResults'),
Finally, let's create a new template to show our results:
1{% extends "blogengine/includes/base.html" %}23 {% load custom_markdown %}45 {% block content %}6 {% if object_list %}7 {% for post in object_list %}8 <div class="post col-md-12">9 <h1><a href="{{ post.get_absolute_url }}">{{ post.title }}</a></h1>10 <h3>{{ post.pub_date }}</h3>11 {{ post.text|custom_markdown }}12 </div>13 {% if post.category %}14 <div class="col-md-12">15 <a href="{{ post.category.get_absolute_url }}"><span class="label label-primary">{{ post.category.name }}</span></a>16 </div>17 {% endif %}18 {% if post.tags %}19 <div class="col-md-12">20 {% for tag in post.tags.all %}21 <a href="{{ tag.get_absolute_url }}"><span class="label label-success">{{ tag.name }}</span></a>22 {% endfor %}23 </div>24 {% endif %}25 {% endfor %}26 {% else %}27 <p>No posts found</p>28 {% endif %}2930 <ul class="pager">31 {% if page_obj.has_previous %}32 <li class="previous"><a href="/search?page={{ page_obj.previous_page_number }}&q={{ search }}">Previous Page</a></li>33 {% endif %}34 {% if page_obj.has_next %}35 <li class="next"><a href="/search?page={{ page_obj.next_page_number }}&q={{ search }}">Next Page</a></li>36 {% endif %}37 </ul>3839 {% endblock %}
Let's run our tests:
1$ python manage.py test2Creating test database for alias 'default'...3...........................4----------------------------------------------------------------------5Ran 27 tests in 6.421s67OK8Destroying test database for alias 'default'...
Don't forget to do a quick sense check to make sure it's all working as expected. Then it's time to commit:
1$ git add blogengine/2$ git commit -m 'Implemented search'
And push up your changes:
1$ git push origin master2$ git push heroku master
And that's the end of this instalment. Please note this particular search solution is quite basic, and if you want something more powerful, you may want to look at Haystack.
As usual, you can get this lesson with git checkout lesson-7 - if you have any problems, the repository should be the first place you look for answers as this is the working code base for the application, and with judicious use of a tool like diff, it's generally pretty easy to track down most issues.
In our next, and final instalment, we'll cover:
- Tidying everything up
- Implementing an XML sitemap for search engines
- Optimising our site
- Using Fabric to make deployment easier
Hope to see you then!