The accepted wisdom in the usability industry is that card sorting is an underutilized tool for Web and software design. But what are the advantages and disadvantages of doing it online? Do you learn as much?

I recently gave Optimal Workshop‘s “OptimalSort” online research tool a spin for a client project to see how it would fare in collecting ideas to a new site design that was in-progress. OptimalSort, part of OptimalWorkshop’s suite of online products (available annually, by the month, or by the survey), promises the ability to get the insight of an in-person card sort study without the time and resources required to run such a study.

Here is the process my client and I went through, and some of the things we encountered about the process of doing an online card sort.

Designing a New Study

Designing a new card sort study entails making choices on: Type of sort, Tasks for Card tasks / names, Online instructions, and Post-survey questions.

1) Type of sort. The client and I chose to do an open card sort (where users choose their own names for categories), rather than a closed card sort (where users sort cards into your existing categories)as we wanted to get fresh input into how the users were thinking about the site content.

OptimalSort made it easy to set up different kinds of card sorts, whether open, closed or a hybrid sort, where you can set one or more categories and then allow users to add their own.

2) Tasks or card names. A set of 20-30 tasks is recommended. After doing an initial study with 28 tasks, we found a high abandonment rate.  We did a closely aligned second study with 21 tasks, so I recommend keeping it to this number unless you’re compensating your users.

While many industry experts recommend using task-based cards (action-oriented, such as “find a news article”), an internal test prior to our larger test showed this was more difficult for users. They preferred the noun-based card names (“news articles”).

OptimalCardSort allows you to bulk import a list of tasks you have set up in Excel, which is a nice feature. One odd bug I noticed in OptimalSort was that it didn’t have an accurate counting mechanism (odd for an automated tool) – sometimes incorrectly reflecting the actual number of tasks in a study.

3) Online Instructions. Any online user study always needs some introductory text for the user as to what to do. OptimalSort comes with basic introductory text, but after doing a dry run with some “friendly” inhouse users, we found it was better to create custom messaging to explain how to use the card sort.

4) Post-survey Questions. Capturing feedback after the survey is useful. OptimalSort lets you add post-survey questions to find out if users have any issues with either your brand, your company, your product, or have feedback on the survey itself.  In our first run with OptimalSort, users were adding their comments to the card categories themselves, so I recommend definitely including one or two post-survey questions to capture this kind of feedback.

Launching a Study

Launching is pretty straightforward on OptimalSort – you hit a button called “Launch Study.” But don’t leave the next screen until you copy the short URL generated for the study –  it isn’t available anywhere else. This appears to be an oversight.

Analyzing Results

Screen Shot 2014-10-07 at 10.22.49 AMOptimalSort’s analysis tools are easy to work with – it walks you through how to analyze results, provided you have some understanding of cluster analysis to begin with (it provides links to outside resources, if you don’t).The built-in Participant Similarity Analysis is worth the price of the tool – it provided excellent insight into the similarities in groupings between participants.

OptimalSort provides a way to export data to Excel or other file formats, if you prefer to analyze results using some industry-standard tools such Donna Spencer’s spreadsheets or the SynCaps Cluster Analysis Software.

We did have problems in combining results from two related studies, however. When we ran an initial study of 28 cards, then another with a sub-set of the same study, we were unable to find a way to combine the two results easily. This can be done in Excel spreadsheets – but not so easy in OptimalSort.  Finally a workaround was reached – we were able to add the results of the 2nd study (with the smaller subset of cards) into the first study by using the manual entry tool. OptimalSort allows the entry of manual card sorts into any study. It took some time to do this, and it did alter the ability to see certain clusters (as there was a big “blank” cluster resulting in the omission of the 7 cards), but our Similarity Matrix was the better for it.

Summing Up

All things considered, I found OptimalSort to be a useful tool for conducting an online card sort study.  It does still have a few bugs and missing features, but it is worth the monthly cost of a study if you’re willing to take the time to fine-tune your instructions. Its best features are the Participant Similarity Analysis (PCA), and Similarity Matrix results graphs. I recommend giving it a spin – trying it for a month (for less than $100) will allow you to do one or more surveys, and add to your knowledge of your users for any upcoming design work.

Share This