SpinachCon 0 Usability Testing Session

See https://fedorahosted.org/hyperkitty/wiki/SpinachConZero

Participants

• 8 participants (1 didn't fill out questionnaire)

• Volunteers in town for LibrePlanet 2014

• Hyperkitty version hyperkitty-0.1.7-1
• Browser - Firefox on F20

Feedback Collected During Test

• Degrades nicely when Javascript disabled; graphs could degrade more gracefully. When JS disabled, they have a loading spinner display that never stops animating. Better to not display anything at all than that.
• Filtering lists on the front page - too many lines appear. If you filter something that has no match, or few matches, you get a screenful of lines. Better to not display the lines.
• Tag system should suggest tags actively used in the system (auto-fill would be nice too)
• For each list on its overview page, there should be a place to read the rules of the list, tag rules, etc. like a forum 'sticky'
• tag system should suggest tags actively used in the system (auto-fill would be nice too)
• list overview should have a place to read the rules of the list, tag rules, etc. like a forum rules 'sticky'
• one user zoomed in the text using firefox. This made breadcrumb expand too large vertically, limiting vertical real estate available to view the screen since the top bar is sticky. It blew out some graphs too, although kpytang's pending pull request will likely fix the graph blowouts.
• A couple of folks after my talk asked about being able to navigate keyboard only, no mouse. We should test for that

• A couple of people - one at SpinachCon, one at LibrePlanet - asked about functionality without javascript. We should test for that.

• Only one user noticed the list popularity filter things on the left side of the screen in the list overview

• BUG: ALPHABETICAL TAB NOT WORKING PROPERLY. IT DOESN'T WORK UP TO "H".
• THE DATE OF THE "LIKE" IS NOT CLEAR ON THE FAVORITES TAB. (user didn't know when he had liked the post. only the date the post was made was listed.)

Tasks

Users were logged into http://lists.stg.fedoraproject.org (running hyperkitty-0.1.7-1 using Fedora mailing list data) and asked to complete the following tasks:

1. You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?
2. Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.
3. You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?
4. In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?
5. Find a post you like an add a tag to it so you can find it later.
6. Start a new discussion on the Fedora users list.

Test Data

Task Notes: By User

Because of some issues we encountered using gedit, we weren't able to collect data in a manner that associated questionnaire users with the user task notes. Oops.

---

User A

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

Anstronomy

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

scm-commits@lists.fedoraproject.org Fedora SCM commits, package-announce@lists.fedoraproject.org Fedora Package

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

Adam Williamson

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

Re: various mishandlings of corrupt GPT [note this answer is wrong]

Find a post you like an add a tag to it so you can find it later.

<No Answer>

Start a new discussion on the Fedora users list.

<No Answer>

---

User B

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

I would rather be over whelmed than under whelmed on a mailing list. More than likely I would sign up for the dev list for the action. Once it got to be too tedious going through excessive email, I would just unsubscribe.

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

389-commits seems to have many discussions going on. Arms-builds has a lot of discussions (600+). Epel package announce and Java sig commits also show higher than average discussions.

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

Adam Williamson, Alexander Todorov, Kevin Kofler or Stephen Gallagher.

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

I had difficulty finding this thread through the link to March 2012. I tried to use the "search" function using "Fedora 17 GPT" and was returned 20 pages of all reference to these topics. I also tried "Fedora 17 GPT March 2012" and the search engine showed no results. I thought that the search would allow me to search March 2012 alone rather than the entire site history.

Find a post you like an add a tag to it so you can find it later.

thought was like/dislike... test admin showed me where tags were

Start a new discussion on the Fedora users list.

done

---

User C

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

Astronomy

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

scm-commits, arm-builds, package announce

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

Adam Wiliamson, Alexander Todorov, Kevin Kofler

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

No. Although this may partly be the slow connection? I searched for the topic but couldn't find a way to search by a specific date. Sorting by earliest or latest date yeilded a lot of results that only showed up ten to a page which was tricky for a popular topic.

Find a post you like an add a tag to it so you can find it later.

I added the tag but I wasn't able to find it later. Maybe I should have chosen a longer tag? (I chose "spicy")

Start a new discussion on the Fedora users list.

If a discussion is the same as a thread, then this one worked!

---

User D

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

astronomy list

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

devel, epel-devel

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

Adam

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

<No Answer>

Find a post you like an add a tag to it so you can find it later.

<No Answer>

Start a new discussion on the Fedora users list.

<No Answer>

---

User E

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

astronomy

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

devel

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

Adam Williamson

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

Nope Flipped throug the first few pages of the threads from March, tried searching for 'Fedora 17 GPT', to no avail.

Find a post you like an add a tag to it so you can find it later.

Done. Bodhi tag added to a thread.

Start a new discussion on the Fedora users list.

Done!

---

User F

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

DEVEL LIST.

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

ARM BUILDS; EPEL-PACKAGE-ANNOUNCE;GOLANG

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

ADAM WILLIAMSON

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

DID NOT SEE THE SEARCH BOX ON TOP OF WINDOW. FIRST THOUGHT WAS TO HIT CRTL+F.

Find a post you like an add a tag to it so you can find it later.

LIKE BUTTON EASY TO FIND.

Start a new discussion on the Fedora users list.

YES, IT WORKED. EASY TO FIND.

---

User G

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

astronomy

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

virt-maint, users, suds

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

John Florian, Adam Williamson

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

• Searched for: GPT and got one result with GPT in the subject then a bunch of stuff that did not seem related.
• Searched for: Fedora 17 and got seeminglg unrelated results, noticed first result was about fedora 20 dated 2014-01-17
• Tried putting quotes around "Fedora 17" - it seemed to help a bit, but still have not found GPT discussion.
• Comment: It would be nice to be able to collapse list. That is to see just the subjects of the messages, without an excerpt from the message, so you would get more messages per page. I got 29 pages of results and did not see GPT within the first couple of pages. I would not continue with this if I was really searching and would probably resort to google. Which is what I did, at google entered this search string: fedora-devel list AND GPT and March AND 2012
• The very first result returned was: GPT and Fedora 17 ..devel-list/2012-March/msg00031.ht...‎ [correct answer]

Find a post you like an add a tag to it so you can find it later.

OK

Start a new discussion on the Fedora users list.

done

---

User H

You've got a crazy inbox and you're really not happy about signing up for high-traffic mailing lists. Based on this, would you rather sign up for the devel list or the astronomy list?

Astronomy

Some of the lists on this system are kind of dead; some have a lot going on. Name some of the lists where there's a lot going on - people actively making discussions.

scm-commits, users, devel

You've heard about Fedora's development list and decided to look into it. Who are you likely to hear from if you read that list?

Adam Williamson (people under most active)

In March 2012, there was a discussion about GPT and Fedora 17. Can you find it?

ry: searched for "GPT 17". "GPT 'Fedora 17'". Ask for advanced search. wanted to search by date restrictions. "gpt 17 march 2012". Found dates listing on RHS of list view. wanted an ability to search but rescritcved to date chosen.

Find a post you like an add a tag to it so you can find it later.

done

Start a new discussion on the Fedora users list.

<No Answer>

User Questionnaires: By Question

Average Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

• Not applicable.
• Better

• <No Answer>

• Somewhat comparable (2)
• Not comparable (2)

One feature I would like to see is...

• <No Answer>

• Clear threading.
• The alphabetical tab working properly.
• Date searching, search by tag, maybe folders. e.g., "personal involvement," "monitor only"
• Small tutorial on query semantics, highlighting of matches.
• More color, larger text, icons (????)
• Ability to sort threads in search results... ability to search within only the selected month

Something I liked was...

• Threading in "View Discussion"
• Nice UI Design, posting online and not just by mail.
• Easy to find buttons (But not the searchbox)
• the little bar graph
• Auto-complete on search
• Easy to find the dates for previous posts
• tagging, +1/-1

If you had trouble performing a particular task, what was the first problem?

• Understanding bar graph
• No by-date advanced search (should have by user, too.)
• Finding the search box
• trying to search by date
• Querying 2 different keywords reduced the relevance
• Search feature can use a little work
• couldn't easily find specific thread in month or search views

What do you think would help this software gain wider adoption?

• Having it installed as default archive interface for mailman archives.
• Just polish and get stable!

• <No Answer> (3)

• a few more ways to sort & search

• Put Miley Cyrus or someone equally stupid up in the corner

Demographics

Age

Gender

Computer Usage

Where multiple selections were made, highest level selected was used below.

---

User Questionnaires: By User

---

User 1

the little bar graph

Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

Not applicable.

One feature I would like to see is...

<No Answer>

Something I liked was...

Threading in "View Discussion"

If you had trouble performing a particular task, what was the first problem?

Understanding bar graph

What do you think would help this software gain wider adoption?

Having it installed as default archive interface for mailman archives.

Demographics

---

User 2

Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

Better

One feature I would like to see is...

Clear threading.

Something I liked was...

Nice UI Design, posting online and not just by mail.

If you had trouble performing a particular task, what was the first problem?

No by-date advanced search (should have by user, too.)

What do you think would help this software gain wider adoption?

Just polish and get stable!

Demographics

---

User 3

Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

<No Answer>

One feature I would like to see is...

The alphabetical tab working properly.

Something I liked was...

Easy to find buttons (But not the searchbox)

If you had trouble performing a particular task, what was the first problem?

Finding the search box

What do you think would help this software gain wider adoption?

<No Answer>

Demographics

---

User 4

Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

Somewhat comparable

One feature I would like to see is...

Date searching, search by tag, maybe folders. e.g., "personal involvement," "monitor only"

Something I liked was...

the little bar graph

If you had trouble performing a particular task, what was the first problem?

trying to search by date

What do you think would help this software gain wider adoption?

a few more ways to sort & search

Demographics

---

User 5

Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

Not comparable

One feature I would like to see is...

Small tutorial on query semantics, highlighting of matches.

Something I liked was...

Auto-complete on search

If you had trouble performing a particular task, what was the first problem?

Querying 2 different keywords reduced the relevance

What do you think would help this software gain wider adoption?

<No Answer>

Demographics

---

User 6

Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

Somewhat comparable

One feature I would like to see is...

More color, larger text, icons (????)

Something I liked was...

Easy to find the dates for previous posts

If you had trouble performing a particular task, what was the first problem?

Search feature can use a little work

What do you think would help this software gain wider adoption?

Put Miley Cyrus or someone equally stupid up in the corner

Demographics

---

User 7

Ratings

On a scale from one to ten, tell us about the software you tested:

Questions

How did it compare to any of the proprietary alternatives, if you have used any?

Not comparable

One feature I would like to see is...

Ability to sort threads in search results... ability to search within only the selected month

Something I liked was...

tagging, +1/-1

If you had trouble performing a particular task, what was the first problem?

couldn't easily find specific thread in month or search views

What do you think would help this software gain wider adoption?

<No Answer>

Demographics