Pagination in Meteor

Meteor‘s a nifty “reactive” Javascript framework, and I’ve been plunking away on it with the help of the fantastic Discover Meteor book.  One thing I don’t quite agree with the book, however, is their Pagination Chapter (paywall, sorry) doesn’t actually implement traditional pagination so much as a variation on infinite pagination. That is, rather than show posts 1-10, then 11-20, their example shows 1-10, then 1-20, and so on.

This is potentially less-than-ideal for a couple of reasons. First, it forces the client to keep a growing number of posts in memory. 10-20 posts probably isn’t a big deal, depending on your app, but if you’re displaying hundreds of photographs (or animated GIFs), this could get pretty annoying. Granted, you can minimize the impact by only loading elements that are visible on screen, but it’s a headache that you don’t have to deal with traditional pagination.

The other potential issue is that, because Meteor is a real-time framework, the server needs to constantly monitor changes to the database and pass along those changes to the client. As the number of documents inside a database query grow, the number of updates being passed back increases as well. I haven’t been using Meteor long enough to know how much of an impact this makes in practice, but updating off-screen data is, at least, a sub-optimal use of bandwidth.

So, how do we work around this?

First, let’s look at the book’s rationale for the infinite pagination approach:

Why are we using an “infinite pagination” approach instead of showing successive pages with 10 posts each, like what Google does for search results? This is actually due to the real-time paradigm embraced by Meteor.

Let’s imagine we are paginating our Posts collection using the Google results pagination pattern, and that we’re currently on page 2, which shows posts 10 to 20. What happens if another users deletes any of the previous 10 posts?

Since our app is real-time, our dataset would change. Post 10 would now become post 9, and drop out of our view, while post 11 would now be in range. The end result would be that the user would suddenly see their posts change for no apparent reason!

Even if we tolerated this UX quirk, traditional pagination is also hard to implement for technical reasons.

Let’s go back to our previous example. We’re publishing posts 10 to 20 from the Posts collection, but how would you find those posts on the client? You can’t pick posts 10 to 20, as there are only ten posts altogether in the client-side data set.

One solution would simply be to publish those 10 posts on the server, and then do a Posts.find()client-side to pick up all published posts.

This works if you only have a single subscription. But what if you start to have more than one post subscription, as we’ll do soon?

Let’s say one subscription asks for posts 10 to 20, and another one for posts 30 to 40. You now have 20 posts loaded client-side in total, with no way of knowing which ones belong to which subscription.

For all these reasons, traditional pagination just doesn’t make much sense when working with Meteor.

Valid points, assuming that we want pagination that’s tied directly to the index of items returned by a particular query — i.e. we want to show items 1-10, then 11-20. However, in most cases, our paginated items are being sorted on some other value (e.g. date) — indeed, pagination doesn’t make a whole lot of sense if you can’t guarantee some sort of consistent sort. And in most cases, our end users don’t care about a particular literal page so much as being able to digest a limited amount of data at a time.

So instead, we can work around the pagination problem by implementing pages based on the value being sorted on. For example, if you’re sorting by most recent date, you would start by showing the 10 most recent items, followed by the 10 most recent items _older than _the oldest item in your first 10 items.

You can see my sample implementation of this kind of pagination on Github.

This avoids the issues highlighted by the book. First, because our pages are tied to a sort value, inserting an item not currently on screen doesn’t shift the items that are on the screen.  That is, if we’re displaying “the 10 most recent items older than December 1, 2014,” inserting something after December 1, 2014 doesn’t change the result of the query (nor does inserting something older than the oldest item on screen).

Second, items that don’t belong on a current page can be easily filtered on the client. The presence of a document on a page isn’t tied to a particular subscription but to its sort value. And since pagination lets us keep only only the current documents in memory on the client, the amount of documents we have to sort client-side (where there are currently no indices) remains small.

That said, there are a handful issues to be aware of with my implementation:

(1) The sort value (or combination of values) has to be unique. If it isn’t unique, and the page break happens to fall between two items with the exact same sort value, then going to the next page will either skip an item (if you choose to show all items greater than that sort value) or show an item twice (if you choose to show all items greater than or equal to that sort value). In my implementation, I address this by using the _id of a tie-breaker when the rank (our sort value) is identical. It makes the code quite a bit messier, but isn’t necessary if you can guarantee uniqueness on the primary sort value.

(2) My code actually requests one more post than is shown on each page. The rationale is to give some indication as to determine whether there are additional values that make it necessary to show the “next” or “back” links.

(3) This still isn’t traditional “pagination” insofar that you can’t directly jump to “Page 5” from “Page 1”. That is, without getting Page 4, we wouldn’t know what value to start Page 5 with (and without Page 3, ditto for Page 4, and so forth). You can step one page forward or back at a time. However, note that you _can _jump by value — i.e. if our sort value is “rank”, we can jump to the “page” with a rank greater than X. In most cases, this behavior may actually be preferable to the end-user (who is thinking in terms of the sort value rather than actual pages).

(4) A corollary to the above is that going backwards can result in some odd behavior from the user’s perspective. Suppose we have 20 items, each ranked in order from 1-20, and we show 10 items per page. The user starts on Page 1, which shows items 1-10. The user then clicks “next” and goes to Page 2, which shows items 11-20. Now suppose that a new item is added with the value 5.5.  When the user goes back from Page 2, the page will still show 10 items, but from 2-10 instead. Going back further will show a single page with Item 1. There’s technically nothing wrong with this approach, but it may be weird to have new “pages” appear before your current page or to have a “Page 1” with less than the maximum number of items on it.

Anyhow, hope that’s helpful.