Usability Test Plan - June 29

Great. Thanks Stefanie . I'll take a closer look next week and provide feedback.

Morgan

Hi Stefanie ,

I've taken a look through the test plan above, and I don't have anything to add at this point. Let me know how we should proceed for next steps. I've included Liam and Subash on this, because they will be responsible for reaching out to ISLG's user to participate in the testing.

Thanks,

Morgan 

That's perfect. In terms of next steps, I'll update and prepare the wireframes for usability testing. Once we think the wires are close, we can start recruiting. 

I'll also put together a screener and share that with you. Typically in the screener, I ask for the participant to check off their availability and we schedule it based on the days/times they select. Will this work for you? I'm not sure if their assistants manage their schedules? I just use Google Forms so we can all have access to the responses and you'll be able to reach back out to them to schedule the time. 

Hi again,
I created the screener so that you know what I'm talking about. This will help us recruit the right proportions of users. 

I think if we aim to begin testing on July 30th that's a very safe bet but it's possible that we could start the week before. I'd like to do a practice round or two with Irit maybe or Paul before we start with our participants? 

Also, do you want to provide an incentive? If so let me know what you think would be best - if there's a kind of gift card that they'd find particularly useful. 

https://docs.google.com/forms/d/e/1FAIpQLSdGzU8tlTgxfFm0X5UjR1iShbzV5amIpX0kbYhyj6bQe_nyRg/viewform?usp=sf_link

Hi Stefanie ,

The plan sounds good to me. Why don't we perform the "warmup" tests with Paul and Chris Thomas, because Irit doesn't usually perform research tasks - or does that matter?

Aiming to begin testing on July 30th sounds good. I'll be away the week of August 6-10, so let's aim to get testing done before I leave. 

Could you change the permissions on the form? I can't access it right now.

Thanks,

Morgan

 

here's a new link: https://docs.google.com/forms/d/1qGOcqWRT3GcNVYWPUrB8FSPSYnKYobJGrI0WxSTDtI8/edit?usp=sharing

I also shared via your email. 

Paul and Chris Thomas would be perfect for the practice runs - we'd want to do those on July 24 or 25, ideally. Those days are completely open for me. 

Do you think 1 week is enough time given the busy schedules of participants? 

Great. Thanks Stefanie .

I've gone through the form, and provided some edits. Note that I created a Doodle poll to schedule the sessions, and then inserted a link in a question in the Contact Details section for people to fill-in their availability. Let me know if you think this is a bad idea.

Paul is available for a session on the 24th, and I'll contact Chris to see if he's available.

The time frame should be fine. Note that I've included the week I'm away, so you can conduct the sessions without me.

Thanks,

Morgan

Great thanks Morgan . My only concern with the doodle is that they may just visit the doodle without submitting the form. Do you mind if we just use a multiple choice grid in the google form itself? I created one for you to see:

Screen Shot 2018-07-06 at 4.18.24 PM.png 57.4 KB • Download

The only other change I made was to the 'no' response - I think we can just send them to a disqualify page, so I added that. 

The only last thing to consider is an NDA. Typically with research sessions, the participants sign an NDA saying that they won't discuss what they see with anyone else. I have one that I always use for research sessions that I can share with you. Typically I send the NDA a couple days in advance of the session asking them to sign or agree to the terms via email. Did you want to do this? 

Looks good Stefanie . The grid option is a good idea. I changed it to a check-box grid, so that participants can select multiple times for the same day.

Yes, please send me a copy of the NDA for me to review.
 
Thanks,

Morgan

UsabilityTest_NDA (1).docx 65.7 KB • Download

Hi Morgan , 
Here's the one I always use - usually for remote sessions, I have them just send agreement over email. 

Hi Stefanie ,

I've done a review of the NDA above, and we'll need to redraft this a little. I'll spends some time on this tomorrow.

Thanks,

Morgan 

Hi Stefanie ,

Attached is the NDA with some revisions. Let me know if this works for you.

Thanks,

Morgan 

ISLG Usability Test NDA.pdf 111 KB • Download

Hi Stefanie and Melissa ,

Thanks for the call today. Here are the questions for usability testing:

Thanks,

Morgan

Full Text Search Usabiity Test Example (10-Jul-18).mp4 2.18 MB • Download

Thanks Morgan  NDA is good to me. Do you mind if they confirm that they've read and agree by email or do you want them to sign and send back. If I schedule the sessions, it's probably easiest for me to send it to them when we schedule. 

Hi Stefanie ,

We'll need something in writing. So they'll need to sign and send you a copy via email. 

Thanks,

Morgan

Ok Morgan sounds good. 

I sent the email to you to maintain formatting. 

I wonder... should we have people enter their work email so that we can verify that they do indeed use ISLG? I don't want to get any random people looking for a free gift card. 

Here's an example for what you could post to twitter: 

We're looking for feedback on early designs of the new ISLG. Would you like to participate in a 1 hour feedback session?  Selected participants will receive a $50 Amazon gift card. To qualify, complete the screener: https://tinyurl.com/ycmbrxsf

Let's not use any hashtags like #participants #study etc. because some people seek those out. 

Also, we should contact those people we talked to before and personally and ask them to participate because that's an automatic 4 people we know would be good. We should still have them quickly complete the screener so we can schedule them along with everyone else. 

Those people were:
Elizabeth ebnicholson@debevoise.com
Diego diego.rueda@freshfields.com
Natalia natalia.zibibbo@freshfields.com
Loukas Mistelis 

Got it. Thanks Stefanie . We'll organize things internally, and I'll let you know once the emails and social media posts get sent out.

Will do on the hashtags and personal emails to the previous participants. Subash and Liam , please take note.

Thanks,

Morgan

Hi Stefanie ,

Just letting you know that we're emailing all our subscribers today, and we've posted about the study on the website, Twitter and LinkedIn. We'll also be posting it in our newsletter this week and next week. Let me know when you start receiving submission for the screener.

Thanks,

Morgan

That's great. Will do. 

Submissions have started by the way.... I expect they'll trickle in. I'll wait until we have a few in there so I can see a cross-section of people and then I'll start emailing to set up the sessions. 

Recruiting is going extremely well! We're getting an amazing range of different user types. I'll probably start emailing to schedule people in later today or tomorrow morning. Were you guys able to reach out to those initial 4? I'd like to reserve 4 spots for them if they're interested. 

That's great to hear Stefanie .

Liam , can you please coordinate with Stefanie on ensuring we schedule the spots for the previous participants you emailed directly.

Thanks,

Morgan 

Hi Stefanie ,

We've heard from one of our clients that they are having problems accessing the screener, because internal security controls prevent them from accessing Google Forms.  The client is a major New York law firm, and I suspect this may not be the only client with this problem. I'm not sure if there's anything we can do about this at this point (except perhaps redoing the screener on SurveyMonkey, or some other platform), but we'll need to take this into account with the responses we're receiving.

Thanks,

Morgan

That's too bad! 

We do have a decent number of people who have already completed the screener - I think we'll end up with more than enough. I do see people in there indicating that they work for a firm, so I think we'll have a good cross section of the various types of users. What I'd say is let's wait to collect a few more and I'll start reaching out. If we start having difficulty recruiting, we can re-create in Survey Monkey. At this point though, we have more than enough who have filled out the screener, especially if we have a few of the others who participated the first time. Let me know what you think. 

Also, did you shorten the URL that was sent to them? Sometimes that will cause it to be blocked. Either way, I think we're fine from a recruiting perspective if you're ok with us not including people from the large firm. 

Hi Stefanie ,

Sounds like a good plan. Hopefully we're able to recruit enough people with the current form.

No, we included the full URL as your provided in your email. The shortened URL was only used for social media.

Thanks,

Morgan

 

Hi Morgan Liam Subash ,
No need to share the screener anymore - we have more than enough people now. I'll leave it open just in case we have people cancel but we don't need to actively get more people. 
Thanks,
Stef 

Sounds good Stefanie . I'll remove the post on our website. Marysia , please remove the note about the testing for this week's newsletter as well.

Thanks,

Morgan 

Well do.

Marysia

Hi Morgan , 
How did you want to handle the gift cards? Did you team want to send them and I'll just send the emails of the participants who have completed a session? 
Thanks,
Stef 

Hi Stefanie ,

Why don't you send me the list of names and emails, and then I'll get them sent at the end of each session after you confirm it was a success. 

How did things go this morning?

Morgan 

Ok, sounds good. This morning went well - she was a student so I think she struggled a lot more because of that, but still lots of good feedback. I'll post the videos on bc if you want to watch them. 

Here's the one completed this morning Ioanna Lapatoura - il93@sussex.ac.uk

Thanks,
Stef 

Yes, she is based in the UK. 

Stefanie , could you confirm the email address again. I noticed that i.lapatoura@sussex.ac.uk is listed in the calendar invite.

Morgan 

Morgan Either email works - the first one is the one she listed in the screener but the second one is the one her email replies come from. 

ok. I guess I'll use the one you gave me. 

Morgan

Completed: Victoria Sahani victoria.sahani@asu.edu (American) 

Great. Thanks Stefanie . Both gift cards for today's sessions have been sent to the participants.

Morgan 

Completed: Nabila Abdul Malik | nabila.abdulmalik@akincilaw.com (She'd like to receive a Canadian Amazon Gift Card because her husband lives here) 

Great. Thanks Stef. I'll send her the gift card now. Session go well? 

Note that I'll join the session at the top of the hour.

Morgan

It was good - she was in Turkey, so her internet was very very slow but once we found some work arounds, it was good. 

Interestingly, she breezed through creating a report. I didn't go all the way to use the words column and filter but I talked slower and broke the task down more. She also had some interesting ideas for arbitrator reports. If you didn't notice, I've been uploading the videos into the UX Research folder at the end of each day. 

Great. Glad to hear things went ok despite the internet issues. A frequent problem we encounter with our training sessions.

I didn't get a notification of the video, but I them now. I'll take a look when I have a chance later today.

Morgan

Completed: Melissa Morton | melissa.morton@international.gc.ca (CAD gift card) 

completed: Belén María Ibañez | ibanez.belenm@gmail.com (US gift card) 

Thanks Stefanie . Great work today!

Morgan 

completed: 
Susanna Kam | susanna.kam@international.gc.ca (CAD)
Mark Klaver | mark.klaver@international.gc.ca (CAD)

Completed mark stadnyk | mark.stadnyk@squirepb.com (american)
David Walker | david.walker@withersworldwide.com (UK)

Completed: Danielle Morris | danielle.morris@wilmerhale.com (american)

Completed: Marci Hoffman (ac) | mhoffman@law.berkeley.edu

Completed: Giovanni Angles (firm) | giovanni.angles@bryancave.com (US)

He was a great participant. He mentioned wanting to participate again or be involved in a beta if you do that. He was very excited about the document comparison, notifications and legal analytics. He mentioned that the notifications feature was already on the wish list he brought to the session today. 

Glad to hear it Stefanie . I'll be on the next call in 15 minutes.

Morgan 

completed: Herb Somers | hsomers@law.gwu.edu

completed: Diego Rueda (USA)
Diego.RUEDA@freshfields.com

Great. Thanks Stefanie . Hope everything went well, and I appreciate you going this on a holiday Monday.

Completed: Claudia Hartleben | claudia.hartleben@arentfox.com (USD)

Yes, Diego was awesome. He brings lots of interesting language perspectives because I think we works strictly in Spanish. Claudia too was awesome - she actively uses the research notepad right now so it was great to have her consideration on that feature as well. 

Great. Glad to hear it Stefanie . Good work on getting all the tests complete.