Web3 User Research 101

[Music] hello everyone hi thanks everyone for coming I'm uh really excited that we've got a sizable crew even though this classroom is pretty much impossible to find I don't know whether anyone else struggled to find it yeah so congratulations you passed the first test you can find uh a classroom in the middle of nowhere um so as you can see this will be all about doing user research or ux research in web 3 we're kind of going to use those terms interchangeably and uh I'm not doing this myself there's another person you wouldn't know because she's hiding right behind me uh and we're going to walk you through this beautiful journey of doing user research Q QR there a code yes so um because it's a workshop you you don't have to only ask your questions um using the the mircat app but feel free to to do that like we are going to check the app and um at the end when we're doing Q&A if there are any questions there we will answer them um yeah so let's get started nope yes these are names hello um just to give you an idea of what this is going to all entail we're going to start with intro and it's already happening so look at us doing great the first kind of half of this Workshop is going to be a crash course in ux research so what it is how to do it what tools to use what not to do that's also an important part of it and then the second half will be an exercise when you can apply all these um facts that that you learned today and then we'll have some time for Q&A all right so um let us tell you a little bit about ourselves uh Mindy please emerge from your cave and tell us a little bit about yourself hi I'm also here my name is Mindy um I've been working in ux since uh 2017 working in web 3 for about 3 years now I am the senior ux contributor at cow Dow we are uh building several products including cow swap which you may have heard of um didn't get any any Applause that's sad um thank [Music] you Cal team are you asleep what's going on uh yeah and then my um my lovely um assistant here not assistant wow co-host yeah so my name is Christina my pronouns are they them and I've been working in crypto since uh 2018 most of that time has been at nosis and then safe this is where we hear thunderous Applause for those companies um mostly as a as a ux researcher and uh I'm based in Berlin as you can tell from how I look all right so now you know a little bit about us but we want to know about to you so yeah uh raise your hand if uh you feel like you are a complete beginner in terms of understanding ux user experience Ono put your hand down okay very few people okay that's really nice raise your hand if you feel like you have like a pretty good understanding of user experience user research what was it can we go back to question one if you're begin oh okay okay back to question one raise your hand if you feel like you're a beginner in ux okay a few more people have decided they feel like beginners raise your hand if you feel like you you understand ux pretty well okay all right pretty good raise your hand if you uh have already done user research on a web 3 product Oh okay nice yes cow guys come on okay good so yeah so the one's already doing research great it's good to see you here as well um um this uh talk is going to mostly be um sort of basic um logistical information how to get started doing user research but I think there will be useful tips for people who are at any level um of doing user research or learning about user research all right so um let's get down to business what is ux research how does one do it all that good stuff so ux research is all about understanding your users or your potential users in order to make better product decisions that's that's the purpose of ux research there are many many ways you can do it and there are kind of two main types there's quantitative and there's qualitative for quantitative you usually need more users for it to be statistically significant 40 is about how many you need for example for a survey to be valid and this can tell you how many people are having a certain problem how much of a problem it is so if you want to quantify certain facts that you already know a little bit about that's that's a good a good type of method to choose there are many ways you can do quantitative research there are surveys card sorting tree testing benchmarking if you have no idea what these words mean don't worry um we're not going to talk about those today spoiler alert then there's the qualitative side so there you can get lots of information from just talking to a few users so five people already provide lots and lots of useful information for you and the kind of questions you might get answers for in terms of qualitative research is why is this happening why are you are users behaving a certain way how do we fix that is this solution suitable how do people feel about it and common methods for qualitative research include user interviews usability testing diary studies field studies again this is not an exhaustive list at all uh people have been really creative in terms of coming up with different ways to conduct research so as you can see this is an impressive list because we have two hours we're going to do these two this is what we're focusing on today and the good news is these are quite easy to learn and you can get a lot of value from from doing this kind of research um without spending too much time on it so let's start with user interviews what they are and how to do those yes so uh user interviews I like to summarize user interviews as um the art of caring intensely for the space of like 30 minutes to an hour uh about how a total stranger thinks and feels so um the way I will um kind of illustrate this is you know how that that feeling that you have when you're falling in love with someone or you know you're really infatuated someone and you just met them and you just want to know everything about them and you just like they you just want to hear everything that they what they that they think and like every sentence out of their mouths so interesting to you that's how you need to feel about your user just for the space of an hour afterwards you don't have to care about them anymore but for the SP of that hour you have to care so intensely about what they're saying that's the mindset that you need to be in in order to do a user interview effectively and um what I mean by that uh is is that user interviews basically are a way of uncovering user needs um uncovering their pain points so that you can optimize your product to fit them so the goal is to learn as much as possible about this person's thoughts fears wishes needs habits um as related to the area the topic of study uh so that you can build a product that will make their lives easier and make them want to use it what do you need to do a user interview uh is basically a list of questions you need a participant obviously um ideally this would be somebody who who is a user of your product or somebody who is a potential user of your product um and then you need a way to take notes or to record conversation uh what can we learn from a user interview specifically so from an interview we'll learn what this person's uh needs and wants are uh what's important to them what people aren't getting from existing Solutions uh that are out there which could potentially be advantages for you uh and then also uh how people make decisions about which products they use um and the trade-offs associated with different using different products in web 3 um now if you sit down to write a list of interview questions uh for a user of your product you might be tempted immediately to ask things like will you use this product uh do you like this product isn't my idea really great um and those are all terrible questions don't ask them ever um the goal is to get people to open up and share anecdotes from their lives this is not like um like a Q&A interview um or you know like a I don't know a survey this is really you want them to be talking more than you're talking um you need to get them engaged in the conversation thinking deeply about their answers and telling you oh yeah and then there was this one time where this happened and that's happened that's when you get really interesting moments from their lives where you're like okay they actually really had a problem here and I think maybe we can solve this problem um so here are some guidelines whoops here are some guidelines um for writing these kinds of questions uh the first one is to ask specific questions general questions get you general answers and you don't need general answers uh when you're building a product you might end up with something like okay so I spoke to 10 users and they all told me that they use exchanges a lot great how does that this help me build my product uh compare that to something more specific like I spoke to 10 people and eight of them used an exchange in the last month to swap meme coins okay that gives you more information about what people are actually doing and then ideally you get the why about that as well and then we're getting into the really interesting specific data that can help you uh Market your product or build your product to to suit a particular audience um you also want to ask open-ended questions as much as possible not yes or no questions at the end of the interview you don't want a list of uh check boxes yes no yes no uh again you want those really detailed stories from their lives um so you really want to ask open-ended questions so then that the participant can take that question and and and start answering and then and then snowball from there and take the the topic to wherever they want to take the topic to wherever is really interesting for them or where they really have a product to vent a project a problem to vent about um another one is don't ask leading questions um so people will pick up if you have some bias uh everybody has bias obviously we try to keep that as low as possible uh when we're doing research but if you ask bias ask questions like um crypto is pretty complicated huh then you already have put that idea in their heads maybe they weren't actually thinking that crypto was that complicated um or maybe they maybe they they they want to you know go along with what you're saying they want you to feel comfortable you know they don't want to stand up against what you're saying so they'll just go along with that um so you instead want to ask a very neutral question like you know maybe how would you rate the complexity of crypto and then see what they say about that uh another is that you want to ask about past experiences not future predictions uh in Psychology we know that people are pretty bad at predicting their future Behavior Uh how many of you have ever sworn that you were going to start going to the gym regularly and then not done it yeah yeah okay everybody here goes to the gym regularly another thing about user interviews that sometimes people lie yeah yeah as we see here uh the point is that people are really bad at predicting what they're going to do in the future they might have intentions to do something but it doesn't mean they're going to do it and they might do things that they had no intention of doing um so it's always better to ask uh about what people have done in the past rather than asking them to predict their own future which nobody can really do um and then the last guideline here is to always ask follow-up questions and this is where we really get into that part of being in love with the user because you know they say something and we don't just say okay yep that's all I needed thanks we continue okay why did you do it that way oh my gosh that's so interesting really would you do it again the same way the next time or or would you do something differently if you could wave a magic wand and like make it different you know make something different about this experience what would you how would you do that can you imagine a better experience can you imagine a worse experience can you compare it with other experiences if you had any of these these follow any follow-up questions that comes to your mind is is useful and it helps make that person feel like what they're saying is important and it makes them want to talk more uh and it gets you really really uh indepth uh qualitative data that you can then uh use in in your product building and uh Christina smooth transition um so that was user interviews now I'm going to talk about usability testing so there are some similarities you are still talking to a person but if you think about what you just heard about user interviews you you really really care deeply about this person you want to know everything you ask them all kinds of questions you're really engaged you're really present there usability testing is a slightly different vibe so with yourability testing you're watching someone fail and you just got to pretend like you don't care at all you do care like let's let's be clear in order to do good ux research you need to have empathy you need to really care about how people feel what their struggles are that's all true but on your face ain't nothing happening so try to cultivate some kind of inner Serenity so some someone stumbles and you don't help them up they ask for directions and you don't provide any it's kind of a struggle at first for some of us but that's really how you get the most valuable insights we'll get into details about why in in a moment but first let's talk about what you need to do usability testing so unlike user interviews that you can just do you know sitting in a cafe with someone for usability testing you need a little bit more of a setup so you need either a live product or a working product when I say working prototype I don't mean it's you know implemented in any way it can be just a series of clickable screens in figma or something like that that's totally fine you also need a list of test scenarios we'll get into that in a moment again a participant it's kind of sad to be just sitting on your own uh trying to imagine things um side note AI is not a participant you need a human participant so um same as with the user interviews it needs to be either someone who's already your user or is similar enough to someone who your user might be and you need a way to take notes or ideally record the screen as the participant uses the product because that's how you're going to really see what they're doing see what they're clicking on see their thought process through the way that their cursor moves that's all super super useful stuff you don't want to miss it and that way you kind of will be able to uncover those problems with your design so that you can improve them in terms of clarity and flow and ease of use so what can you learn from usability testing you can see what it is is about your product or someone else's product can be a competitor that makes people struggle or get confused or make mistakes so this is where this stumbling thing comes in that's actually good like you want to watch people struggle because if someone does everything in three minutes and has absolutely no issues I guess it's good because you found the product is easy to use but you don't get any insights in terms of what you can improve and your goal is is to come away from the test knowing okay this screen is confusing here the button is hard to find um the text makes people think that the feature is not what it actually is all of those things so it kind of counter counterintuitively you want people to struggle um but you can also learn what people like about the product they might just say oh wow like this this really makes a lot of sense I was waiting for this feature I've I've heard that many times it always is super nice to see people excited about something that you're building um and also you can get ideas about what else to build so how to build maybe the V2 of the of the feature that you're thinking about um so all of these you can uncover so what do you need to have in order to do usability testing you need to get your participant to navigate the UI and if you're doing it online which I think is how we mostly would do it how hell hello yes right um let's maybe exchange this one thank thank you perfect um where was I yes so if you're doing the test remotely have the tester share their screen so that you can see what they're doing um because that's that's where you get all those valuable insights and ask the participant to accomplish some some task maybe task is not even the right word it's more like a scenario more on that very soon again it's important for you not to interfere as much as you possibly can do not nudge the tester to a certain correct path do not guide them as much as you can if you see someone really really struggling you can you can ask but wait just a little bit longer even though you you're like oh this is so obvious you you you if you just like clicked on this button you would get everything you need don't do it don't do it observe what the user does because that's how you uncover what your users are likely to do in the real world and see how they solve the problem and that will help you understand how to improve so unlike user interviews where you can just kind of go right ahead and start interviewing the person before you do usability testing you need to prepare the person for how that's going to work because it's slightly unusual for someone to just sit in front of you or even in front of their computer and do this kind of fake is scenario and and pretend that they're accomplishing something so you need to get them in the right mindset um tell them to just think out loud and be completely unfiltered if they like something if they don't like something they should just tell you everything and tell them to even pretend like you're not there not in the room or not on the other end of the zoom call just your goal is to have them act as naturally as possible it's hard and there's always going to be an element of um awareness of being observed but as much as you can encourage them not to do that you should do it and remind them that you're testing the UI so the Prototype the product whatever it is and not them this is very important because people get self-conscious people start worrying that they're making mistakes people are starting to think oh no I messed this up this person is going to think that I'm stupid that I don't understand things but I'm so smart I've I've been working in this industry forever how do I not get this people get in their head that's normal that's why it's important to remind them this is not about you this is about the the UI uh if there if something about it sucks we'll fix it don't worry and also you you can even tell them I I didn't even design this so you're not going to hurt my feelings if you hate it even if you did Design This say that you didn't because that just puts people at ease about criticizing something not everyone is comfortable with that and uh if it's a prototype make sure to mention that it's a prototype because some of them are pretty highdea and people assume that they can just click on everything most of the time that's not the case make sure people know that okay so now we get to the tasks things so as I alluded to don't think of them as tasks because tasks are kind of efficient and that's not your goal your goal is to get someone to behave naturally and people aren't always very efficient the way they do things naturally so how to write out a test scenario choose a flow or a feature that you want to test for example it's a really really important flow for for your product so if you're just starting with with testing can be I don't know for a wallet sending money checking balances connecting to a dab stuff like like that um it can be a flow that you know are is a particularly tricky one or really really risky one it can be a flow or feature that is new so once you kind of get into the rhythm of doing ux research um when there's a new feature coming up that's a great a great moment to to do um the testing because that way you uncover problems before the feature is implemented and you change the design in advance so you save time or you can test changes to to existing flows that you know users already comfortable about but it might be different so you want to make sure that it stays comfortable for them so as I mentioned scenarios are not tasks so don't say okay so now click on the sign up button and fill out the fields with your information that's not a scenario that's an instruction and the goal of a scenario is to give the Tester the minimum absolute minimum of the required information and learn how they approach the the the scenario how they would do it given their own understanding their own assumption their own habits so for example if you're testing an exchange you can say Okay imagine it's often good to start the test SC with imagine imagine your favorite token just dropped 20% you want to take advantage of this opportunity to buy more how would you do that so you see it's it's kind of phrased more like a question like a hypothetical scenario and that invites your tester to do whatever it is most natural for them to do and then you learn about how they would do it in your actual product or if it's a wallet you can say hey imagine you made a new friend at Devcon stayed out all night talking about ZK proofs now you need to pay him back for for dinner how would you do that and then you sit back and observe there will be struggles in usability testing that's kind of the name of the game as I've been telling you and there are different ways to handle those struggles so for example the disp can ask you a question but not just like hey what's your favorite color um it's like how do I do this or what do you want me to to click on or how where do you want me to navigate to or if I do XYZ what will happen this is the moment for you to tap into that inner Serenity and be like that's a great question I'll answer it at the end but for now just do whatever feels right I want to learn about your assumptions your habits what you would do if you have any questions at the very end of the test I'll answer all of them if the participant seems confused which most likely they will be um you can kind of nudge them to explain a little bit more about what is causing the confusion so you can ask what do you think is happening like I I see I see that you're wondering about something I see that something is giving you pause what do you think is the issue how would you solve this problem because sometimes especially if it's a prototype you you can't really do everything and people normally would I don't know close the app open stack exchange ask a question there or like Google something so it's good to also give them space to to tell you about that and so if the participant is really really really struggling so they're about to lose all of their money um rage quit the the test and you can see that they're just not getting anywhere you can say all right so I think I I got the information that I need would you like some help don't say the thing that needs to happen right away unless again like they're about to click on the button that is going to destroy their life forever um unless that's happening just ask would you like some help and then let let the person tell you or you can just say okay let's let's pause with this one and go to a different flow do something else so this was the overview of usability testing um but these principles apply to just doing ux generally so studies show this not us saying this people have looked into this and the study shows that for usability testing just five participants is enough to uncover about 85% of the issues as long as those participants go through the same flow so if you're giving people the app and saying oh can you use it for a bit and and do something um then no five people aren't aren't enough but as we said before you need to be specific with your scenarios with your questions so as long as you're specific and and you give the same scenario to everyone five participants is enough to uncover a lot of the issues that being said you need to make sure that you're testing with the right people if you go out into the world and your product is geared towards protocol Developers for example and you get five casual coinbase users that like have heard about metamask your insights will not be relevant to what you're building you need to make sure that you test with the right users and finally we love ux research like that's why we're here but it can't answer every single question that you will ever have it will tell you about user needs habits motivations pain points that's a lot but it cannot tell you for sure whether your product will be successful or whether a particular feature will be actively used by everyone that's just there are some limitations to what you can accomplish with ux research as you're uh as you're going into uh starting your your your first research um studies or continuing the research studies maybe that you're already doing um it's important to keep in mind especially in web 3 that web 3 users are very privacy conscious um as they should be um they're often very suspicious especially of things that sound too good to be true of offers for money in exchange for doing something of random people sending them messages on random platforms uh again as we all should be um so it's very very important it's important in any area of uh of research even if you were doing e-commerce or something like that but it's especially important in web 3 that you give users as much information as you can upfront that you're very upfront about what it is you plan to do what it is you plan to ask of them what it is you plan to do with their data um and that you keep the requirements to join your study at an absolute minimum um so some examples uh the first thing you want to do is you want to inform somebody when you're when you are approaching them or when you're putting out a call for for um to recruit participants um you want to inform people what's going to be involved are you going to require them to show their their faces on video are you going to require them to speak um what language are you going to require them to speak and do they have to speak perfect English um do they have to share their real name do they have to share their real wallet address do they have to share their screen do they have to connect their wallet to adap uh do they have to do a transaction with their wallet do they have to uh will you take a recording how will you where you will store that recording um all of these things uh are things that people need to be informed about upfront so they can make that informed decision about whether it's worth it to join your uh your study and so then it's best for you if you have kept those requirements at a minimum so that then people feel like okay I've been told that if I join this study I don't have to connect my wallet to any dap and so if I believe that then there's very low risk for me right to join this study okay and then you get more likely to get participants if you expect people to you know just hop on a Discord call with you and share their real wallet where they have all their money and uh you know connect to your your dap that's probably a hard sale um also make it available that people can ask uh to request deletion uh of their data that you've taken from them of the notes that you've taken from their study after a certain point um let users give consent granularly so ask them not just yeah these are my requirements do you consent to these uh what I typically do is say hey I would like to take uh I would like you to share your screen I would like to take a recording of that screen and of your audio um but you can consent to these separately so you can consent and say yeah I'll join your study and you can watch my screen but you can't record it let people consent granularly to each different thing that they're willing to share um and that also makes them feel a lot more comfortable um joining your your your session um and again keep it Loose if possible somebody says no I really don't want you to take notes or I mean I really don't want you to record um then just kind of be willing to say if if if possible we be willing to say like yep okay you know what that's fine I'll take notes by hand if you're willing to kind of slow down and speak slowly so I can take those notes and people will be um understanding of that um another is that uh if people are going to be doing a transaction for for this test um you need to provide them with gas if they're going to be swapping funds for this test you need to provide them with the funds for swapping never expect people to use their own funds to join this uh this this test of your product um that's not ethical and um you know it's just also kind of a bad sell It's A Hard Sell so yeah always make sure that if you're providing funds also provide gas funds as well if they're necessary um if they're sharing their screen uh let them know that they only have to share um the tab that your uh that your dap is in so a lot of people will just automatically share their entire screen which means if they switch over to another tab if they open another app if they open their metamask you can see all that you can see what's in their metamask they probably don't want to share that with you so guide them to only share the tab that is relevant for you um and then uh yeah keep info sharing requirements uh to a minimum tell them they don't have to share their name tell them that they can refuse to answer a question uh if they don't want to answer that question uh phrase questions in ways like would you be comfortable telling me or would you mind sharing you know many the size of of a transaction how many do dollars you trade in a typical transaction some people don't want to share that some people are fine with it it's best if you if you phrase it in a way that lets them know that they can say no I don't feel comfortable sharing that and that's fine um and then uh again in any area of user research the number one Golden Rule is always always always compensate your user research participants they are sharing their time with you they are sharing their knowledge they are sharing a piece of sort of their daily uh behaviors their their thoughts with you this is super valuable information uh especially in web 3 where again there's a lot of emphasis on the importance of data and how we're all uh we all have the right to our own private data right so this is extremely valuable information they're sharing with you um they're doing it so that you can build a better product um and the absolute number one thing that you have to do is make sure that they are compensated uh for their time whether that's crypto whether it's you know some um other perk that you can offer them related to your product or whatever make sure that it's something that's worth it for them I think this is your area of expertise I mean it it can be I think I know what I'm talking about I don't know whether you can tell um so let's talk about tools so all these things you need to accomplish them somehow um and we've got some recommendations you don't have to follow them exactly you can also do do your own research see what works for you but just so that you can have a certain idea of what we personally prefer here here are some suggestions so in terms of recruiting users that is quite tricky in crypto again because of privacy considerations because a lot of people are kind of anon um but there are plenty of options so if your project has a Discord you can use it to recruit your user so the way I sometimes do it is that if I'm researching a particular feature for safe I search the feature name in Discord and see what people have asked questions about it and that way I know okay this person cares enough about this feature to ask a question about it they are probably using it at least a little bit they what would be a good fit for for the test you can do a popup in your dab and just encourage people to sign up to do to do research you can try a dedicated recruitment platform there is even a crypto specific recruitment platform called despark you can try that out you can also try out General user research um platforms but they are very unlikely to have advanced crypto users so most most of the time you would get people who you know have some crypto on on coinbase have maybe a tiny bit of experience with metamask but nothing say nothing more advanced than that um you can recruit here like there are plenty of people and you can get to know someone ask them what their struggles are ask them what they're using in terms of wallets or exchanges or whatever the product is that that you're working on and um if they seem like a fit you can ask them whether they'd be willing to to talk to you um about your product for getting consent uh a survey tool usually works quite well so I can do Google forms or type form or Survey Monkey whatever it is that is easiest for you to get a hold of totally fine for conducting user interviews or tests Zoom is is a good option because most of the people know about it they're familiar with it they have it set up often and you can easily record the call for scheduling you can use use calendly that's that's a nice option but whatever zoom's internal schedule or Google Calendar that also works for keeping track of your tests and that seems like an unusual thing to mention but trust us you want to keep track of what you're doing because at first you'll be able to remember and then fast forward half a year and you're like did we test this feature did how many people how many people saw this did we did we talk to this guy he I remember his name but did I pay him how much did I pay him I don't remember keep track um so note who you talk to when about what what their contact info is whether you've paid them and the spreadsheet tool like I don't know Google Sheets notion air table any of those are more than suitable and then for analyzing your notes and recordings and kind of storing them somewhere uh is I would say the first area where you might want to get a specialist tool so all the other ones I've mentioned so far you're probably already familiar with you've used them for other non- ux research e purposes for storing and analyzing notes and recordings it's it's probably worth it to at least look into a specialist tool um both Mindy and I use duvail and it's not the only one at all out there but it's quite easy to learn and it has a free version that has plenty of features that um should cover you if you're just starting all right we've been talking for a very long time and you're probably sick of our voices um we're going to talk some more but now it's going to be different [Laughter] [Applause] it's not over it's not over I'm glad that someone was like oh my God yeah they're gonna stop talking we're GNA be talking but we're gonna be we're gonna be talking directly to you okay that was yeah oh no never mind I I didn't like it so I so I gave up immediately um so no I do want to ask a question so how many of you have a a project that you're working on right now that you would like to do user research with okay excellent perfect so what we're going to do now we're going to do a group exercise um I'm going to let you group up with people at your table you can group up with like the whole table or just half the table or you know talk to people it's a conference it has the word confer in it you know like talk to other people um so you know and this is a workshop so you can Workshop things with people Workshop things with people are leaving they're like no I can't talk to people all the antisocial people are leaving it's fine I would be leaving too if I weren't up here ex oh okay we have we have a few antisocial people who are staying y'all are the real MVPs give it up uh yeah so I will ask you again to find a few people at your table that you can do a a little bit of work with um together with your group uh you will pick a project it can be one of your projects that you're working on uh one of you at the table is working on uh or we will give you an example project uh that you can just sort of imagine that you you're working on uh and we will ask you to create a research plan uh for that project so that by the time you leave here today you will have a ready too research plan and you could even start doing uh research on this project here at Devcon um with uh with potentially real users um yeah so where's the clicker here's how it's going to work uh first I'll ask you to again to pick a project that you want to work on our project that we're going to give you or a project of someone at your table then we'll ask you to choose uh one of the two methods that we explain today usability tests or user interviews whichever whichever one seems more relevant for the project um and then we will ask you to uh make a research plan and Christina will be handing out um the guidelines here for uh how to do this as uh as they hand those out I'm going to be uh explaining a little bit more about what is put is is is in this research plan and I got to do the clicker as well so um first you'll need to think about well first you'll need to think about what is my project which method are we doing then you need to think about what are the goals of this study what do we aim to learn from this research is it you know people's General thoughts about um this area maybe if you're working on a on a DA project and you want to learn people's General thoughts and like experiences with AI uh maybe it's more specifically like I have a project I have a prototype I have a UI and I want to know where people are struggling to use it um yeah so like what are your typical goals um what are the criteria uh what how will you recruit users what how uh what will you have users consent to how will you get their consent how will you compensate the participants again very important uh and then as well to plan out the research uh the research interviews um and Christina and I will be walking around as you go ahead and get to work and answering your questions during this process can I just STI it like that okay cool huh um if if you need to refer back to any of the the examples that we gave any of the information that we shared during this talk uh you can scan this QR code and uh find all that information uh in a medium article as well to help you out think okay is anyone missing these or wants an extra copy okay one here one here cool coming up if anyone wants more at any point just let me know e [Music] e for B tell [Music] TR e [Music] e e [Music] t up [Music] I'm e t alk know e e to video [Music] e e sure [Music] go ask oh [Music] B [Music] Bo o [Music] want [Music] you oh [Music] it's [Music] B hey so in about 10 minutes we'd love for some of you to share what you have been thinking about like things you have come up with things you maybe still have questions about things you're you found trickiest what you've discussed um so let's take another 10 minutes and then we'd love to hear from some of you [Music] f e [Music] want [Music] you [Music] w your [Music] think sh [Music] you [Music] s the o [Music] sh [Music] e e [Music] y all right all right so we are really really hello everyone Hi hi hi we're really really curious to hear what some of you came up with like what products you decided to to research what research plans you've you've come up with um would anyone be willing to share um what they what they've discussed so far and and how that went it it like this is not a quiz we're not going to you know grade you on how well you did but if you want some feedback we'd be happy to to provide some um we're just curious how this how this has been going for you anyone willing to to share just a little bit yay thank you so much um I did both I did like brief in interviews and then let the users test out the product I that was not the [Laughter] activity well I executed the plan so it was really insightful for me because first of all what I really liked is that I had like two different culture users so because that was really interesting to me what differences they could see on my product no and I asked about uh first of all just so you know my product is an nft platform that donates funds to Environmental Conservation projects so I got a a really good Insight of like what are the incentives that people have behind purchasing nfts we also did um a lot of technical issues that I hadn't seen before I could see and also like re I I I had a good Insight that like foreigners are interested in Latin American Art because it's different it's not like something that they us usually see on on other nfts so I thought that was really cool and yeah thank you so much both of you for helping me and that's it yeah thank you I I like that you mentioned that you uncovered some technical issues that's kind of the unintended bonus often of usability testing you just you just find a lot of bugs and often after usability test I I have to write to some of our devs being like oh also this happened do you know why and they' be like oh I've never seen this interesting um so devs devs like when user research happens as well because then you find a bunch of bugs for them to squash yeah for a certain definition of like cool thank you so much for sharing thank [Applause] you um does anyone else want to share yay awesome test test hello everyone hi my name is tiha I'm a ux designer so uh the project that we're going for is the beta Basque example project so we're going with uh user interview as a research method and our research goal is to understand what problem users have when creating a wallet and for recruitment our target audience is basically novice users basic crypto knowledge but uh crypto Enthusiast and recruitment what does it say here uh okay how will we recruit them we have a few options here um conference web three events uh Reddit forums and Twitter and as for consent how will we consent participants since we're using um inperson user interviews uh we'll ask them to consent for voice recording note taking uh their time and also if they're pre they prefer a one-on-one session or um interview with a lot more people so can you uh tell me about that with one one person or or many people yeah one-on-one if they prefer um a one-on-one session like a private session or if they're open to um talking to sharing their um details with like other people oh oh I see so yeah for example if we're doing it at a conference or events there might be people around so yeah yeah yeah that makes sense and then for compensation uh we're going with a few options uh either our own token nft access to side events yeah swags or airdrop yeah yeah all of these are are great yeah so for um the interview itself uh the duration will be between 15 to 20 minutes yeah and so these are the questions that we would ask the first one is um what are your goals getting into crypto second which digital assets are you interested in and then um tell me about a recent weet free product that you use and why did you choose to use that specific product and yeah that that last one is is a really really good one because you kind of asking people to just tell you about their recent experience recency is good because they still have a lot of memories about what just happened yeah and then it's it's specific so that they don't have to imagine things or lie to you just because they want to fill the space you know so and then this is kind of like a a follow-up question to that question briefly explain recent problem that you have with that weet pre product and what do you wish they would have done better yeah that's all from us thank you thank you any else oh over here hello I'm Hussein uh founder or developer and developer of Campus AR which is a decentralized learning management system s for companies and communities uh we are the first uh LMS on U chain uh and uh we chose usability test and uh there will be five scenarios and for each scenarios we will provide them five or 10 minutes and actually uh I will use this um plan for for my friends I will start my friends in school so I I will um ask them like uh follow the most Basics basic flows uh for example registrating verifying your identity and our course modules registrating Arc modules because it's a bit com maybe a bit complicated for non- crypto users because to register an AR module you need to stake a sort of token that's determined by the publisher of the arc module so um CR criteria is uh being students Community or company member of company or Community um so yeah I I've written two uh scenarios for now but we'll complete them first um onboarding process explore the onboarding process try to First verify your identity with open campus ID on chain verification using your wallet or email it's important it makes so easier uh to onboard uh for non crypto users and then uh here are the required G funds let's try to register for an arc module and then third one maybe uh go through the arc module experience and tell me uh the difficulties or the good parts you have experienced uh fourth maybe uh in each Arc modules there are some parts that you collaborate some people you give some feedbacks their codes they give feedback backs to your Cotes and uh what's the general it's yeah it's it's a a bad question what's the general experience you have uh I will make them specific it's maybe fourth one and um fifth one maybe um in campus Arc you can register some Talent programs called Arc Talent programs that's um uh organized by by web tree companies to explore the best talents these programs uh include some Arc modules to complete uh each Arc modules have three or four Pro projects to complete so uh the whole idea of the arc Talent program uh to make them make it easier to explore talents uh like um there there's a proof right uh completing uh many project that's determined by the company itself right so I will uh ask them to join uh AR mode uh arct programs and ask some specific experience details so uh it's all thank you thank you one thing I will say is that um it can be really hard doing usability testing with your friends because it's even harder to pretend like you can't help them or even harder for them to pretend like you're not there so uh you really have to tell them in advance like listen during this test I'm not your friend yeah I will definitely say actually uh I'm planning to um like open the uh screen record and leave them by theirself so uh yeah uh I will get some consent about sure okay it's an important Point by the way thank you I also wanted to ask that uh add that maybe during all five scenarios is a bit ambitious you you'll be surprised how long it sometimes takes people to do what you think of as basic tasks so definitely like try going through everything that you want your testers to do MH um and and time yourself MH and if it takes you someone who knows the flows a while budget for maybe 3 to five times longer uh for for your testers so it might be necessary for you to prioritize a little bit and be like okay all five will take a while and and usability testing is quite tiring both for you as as the moderator and for someone who's like oh my God oh my God what do I do I don't understand anything um so try to yeah I want to pick pick the ones that are most important just focus on those yeah I want to all of them in one session yeah you're absolutely right perfect thank you thank you who else wants to we already have someone with a with a mic oh excent hey uh hi guys I'm PR uh uh I had a big team here uh um so I have uh a few designers here uh and deel and there was this D guy so uh mix bag of Team uh mix bag um uh so um so we have an alpha product uh we we're from 42 money and uh we basically uh trying to build decentralized front ends for smart contracts and uh um we really haven't uh figured about um who the users are currently like we have like we are an impass where we can you know uh go into different directions so we're still trying to figure that out so we started with usability uh testing but then we F to uh user interviews and uh what we found was that uh initially um the users were hackers def analysts right uh they could um you know like just prompt the llm to be able to generate front ends for smart contracts like directly interact with smart contracts right so found a lot of useful things uh from this uh from this exercise uh for example D finalists need a little more um interactivity for with drag and drop uh they need a little a little more interactivity with uh drag and drop options like components uh they need a way to be able to research um where the yields are coming from you know like def Lama maybe or something like that and uh the ux guys uh the they had suggestions about um you know having the Twitter feed of um the of the smart contract that you want to interact with as social proof as what is exactly going on right so and uh data about the smart contract is also useful uh you know like security and uh you know risk about the interacting with the smart contract and then uh generating the fundr for it so yeah so basically found out a bunch of things uh useful and uh yeah cool thank you anyone else wants to share yay uh hello one uh we did a tiny little bit of usability testing here so oh yeah you also just went ahead and and did the thing we we just did the thing we just did the thing so um so uh for you to understand the context um I work at UH graphin which is what we do we do an alternative Android if you ever used Android there are multiple different types of Android and we have our own type of Android and so very recently very recently I helped our project uh to work on the setup wizard thing if you ever heard about it Setup Wizard is a very first thing a user interacts with when they uh open up the phone so and uh we've decided to do a tiny little bit of user experience testing and thank you very much for doing that uh because we uh my friendly colleague uh has never ever used an Android phone as far as I understand right used Android okay yeah long time ago long time ago yeah yeah very long time ago so yeah no it's really good because uh it's amazing to see uh people with different uh perspectives on six and so uh here are a few things that um I can say about our setup Wier for example um when the user first like sets up the fingerprint uh for their phone um it's usually a good idea to show them like a like an animation right to show whether sensor is but it's still not clear and the animation might not be clear and uh or or other stuff so like not not not everybody knows every single phone not everybody knows where all the sensors are or what kind of sensors are so it's very complicated and so if you never ever like so oh I'm so sorry I'm jumping from place to place yeah yeah but like uh it's one of the one of the things that we'll work on more is to show a better fingerprint animation so that the user would know where the uh fingerprint sensor is located yeah this is one of the things and I think I should probably stop now because otherwise I will talk a lot more and this is like the first part of like my two pages so yeah thank you thank you that was really funk [Applause] anyone else yes hey hi um so we had a lot of help from the guys here uh actually we uh we have a game we would like to create and uh where they pinpointed the first uh thing that that maybe if we think something is an issue or a need for users we we should we should ask them so for example we thought that that that uh um um um ownership of assets digital assets is a is a key question and it turned out even in this table it's not a key question for everyone even though they play so uh yeah that's maybe the colleague here phrased very correctly that we started too technical so we started that we must implement nfts we must implement blockchain we must Implement our own coin or whatever or own economy and it's not sure we have to so yeah I think I uh I learned and it's a back to the basics so thanks guys thank you I just want to it was useful on the motivation of the game H they they are building a web three game H so they could own the the Collectibles so we wanted to get users who experience spending a lot of time getting Collectibles and H losing them because the company goes out of business uh I don't know if you want to add something dim Di and money okay thanks that's I guess that's all thank you thanks um anyone else wants to share [Music] anything I think do we have any questions on uh on M so apparently you know how at the beginning those those of you who are here from the very beginning uh you know how we told you that there would be a QR code for you to ask questions well that was a little bit of a lie there was a QR code and you could submit questions but apparently they didn't go to the place that we could actually see so if anyone submitted questions it's uh too bad because we didn't see them yes you have a question yes please go for it okay I have a question so uh you guys said that it is important to um find the right customer to interview with so how to find the right Customer because uh maybe we uh choose randomly five customer maybe they just a small group of our total customer so uh uh do do we have any rule or do we have any uh anything to Define out the right customer I can start so um the five person rule is for usability testing the reason why it typically works for usability testing is because um usability testing is very much based on like psychologically how a person understands a UI how they react to certain things um and so people who are in Fairly similar buckets will T typically have the same reactions and the same understanding of those things so if these people are already your users and you do a usability test with them it's to typically five people is is okay um so that's the first thing now if you are looking at um a larger group like a very large user base and you want to do like user interviews uh or if you're just choosing people off the street or choosing people at Devcon you probably want to write some criteria and that depends on really on your business um like what what what you're trying to do for your business so if you're building a game and you're trying to learn um you know why some people don't play web 3 games then you want to look for people who don't play web 3 games but who do use web 3 but do use web 3 like why are these people doing web 3 but they're not doing um you know web 3 games or if you are if you have a large you know like I I can talk about you know my my um the Dow that I contribute to um we have uh we have um an exchange platform for example and many different people go and do swaps on the exchange so when we're doing user research we have to think about okay um if it's for a specific feature then we want to talk to maybe people who use that feature if it's for you know if we're interested in how to improve the product for people who do very small trades or for people who do very large trades we need to look at the data and try to find the users that do those kinds of Trades um if we're interested in going on a different chain if we're adding a new chain maybe we're going to add base and we want to like understand stand base users then we need to find people who do like web3 users who do swaps on an exchange uh and use base so like the criteria can add you know depending on how specific you need to be for the particular project prob something to add I I think you you covered most of it the only thing I would add is that there is essentially no way to avoid some element of Randomness like you said if if you just pick five of your user isn't that kind of random and you should have criteria you should be aware of okay what exactly you want to test or interview about and what kind of users care about this the most or could care about this the most but if you already done that you've defined your criteria you've searched for those particular users there will be more than five most likely that fit those and there there is an element of Randomness and that's okay yeah user research is not an exact science do you mean I mean it's even in you know especially when we're talking about qualitative research when if you're talking about quantitative research and you're trying to get a really statistically significant sample and like you know look at large amounts of data then things are different but with qualitative research it's it's qualitative it's subjective it's not an exact science and some things will be random and some things won't exactly fit and that's just kind of part of the joy of uh of doing this work actually you have to be comfortable with um with things not always fitting together perfectly yeah does that answer your question yeah perfect thank you anyone else yes U hi um how to create user research on a product which primary focus is not the UI but rather providing some functionalities like imagine AWS without the UI everything that it provides but without the UI well so what can you give um I mean when I said UI without shiny buttons like you can still use the terminal you can still call some apis basically you can still use the product but there are nothing like the login with I don't know buttons for input Fields it's basically pure the functionality you mean how to conduct resarch for a that's yeah yeah yeah yeah so how to conduct a research for for a toolkit for example well again it depends on what kind of research we're talking if it's a if it's an interview then you don't need the UI at all anyway um and if you want to do some kind of usability testing you actually don't necessarily need um you know your typical like login page for a to be usability testing some people do usability testing with you know vacuum cleaners uh we're just we happen to to work in in an environment where everything kind of has a certain number of familiar elements but that's just because uh we're all kind of in this same context but you can do visibility research on like on on this chair if you wanted to as long as you ask someone to use use the product in whatever way that makes sense like using the terminal or coding or using the developer documentation anything that that might be involved in that um you will still get information on okay this is um a struggle this is where someone gets confused they have a certain expectation and they think that something will happen and it doesn't or they're like oh I wish I could also do this so You' still get all of that valuable information even if there's no like sign up is it realistic and you sit with the people yeah that's what I wanted to ask next like is it realistic to do it live or just tell them okay when you are done with using it checking the documentation and everything else just please then provide the answer on so so maybe you can think about what what do you care like it's like H that they can integrate easily your product that they think is a great product like uh which problem do you want to solve for them H maybe if it's a market for that uh once you have that clear maybe you you can design an exercise with these people or like or like a survey uh with the target group where you would gather this information does it make no I mean the first part is perfectly clear like okay you know what you need to do but then when you when you create the product like that part of testing because I mean you mentioned videos audio that I mean that last for an hour or two but basically to read through the docks to try it out it takes much more time so I'm not sure if can like can we do it just sit here this is the laptop please do it I don't know two three 4 hours I mean do the people test the product live for 4 hours or not I don't know or basically tell them okay please do this whenever it's comfortable comfortable for you and provide us the feedback sometime in the future so I would say that um if you're doing feedback afterwards then an interview is actually a good way to do it because often you know you send out a survey like a feedback form people don't like forms people really really don't like filling out forms and if they can avoid doing that they probably will especially if they need to like write texts oh forget it um so here you really might want to get those people um as close to the moment that they did use the product as possible and then ask them okay think about the the moment that that you used our our product let's say yesterday um was there anything in particular particular that you struggled with did you notice anything missing um what was the point where you were like oh they have this thing isn't that nice was there anything that surprised you uh in a positive way so kind of just try to get people to reflect on that experience and and that might be a better option because sitting somewhere with a person for four hours it it can be done but it's kind of an intense thing to do um so maybe not necessarily the the right way to to start you know yeah okay thank you thank you um and I think we are incredibly out of time we have one minute left so anybody has a quick question yes oh no two people whatever I mean we we'll be still we'll be still here like outside of of the of the classroom if you want to talk to us afterwards but uh yeah if if someone wants to do a super quick question then yes yeah uh this will be super quick yeah so basically we at an impass where we're trying to figure out uh with the product uh where we like we have like sectors of users which we want to Target and uh um if you want to Target a specific sector then uh we need to build the product a little differently right and uh um we're trying to decide which uh paths to take from uh user experience and user design like how do we how do we you're so sneaky this is not a quick question this is not a question we will we will answer it outside of the classroom thank you so much to everyone who who came and stayed