– Hi, I’m Joe Welinske, and I am the program manager for ConveyUX, and that’s been Seattle’s annual user experience conference for the past seven years. It’s produced by Blink, and we’re getting ready for our eighth year. The conference is coming to downtown Seattle March 3rd, 4th, and 5th, and we’re gonna have more speakers than we ever had before, more sessions, full three days. And one of the fun things I get to do is talk to our many presenters before they get here to Seattle. And so today I am talking with Claire Pacheco. Hello Claire, how are you doin’ today?
– Hi Joe, great, thank you.
– I am talking to you from Blink’s downtown Seattle office. Where are you talking to us from?
– I’m actually in Toronto, I’m calling from the Toronto office in, PagerDuty’s Toronto office. It’s a lovely new space, we just moved in about a year ago. And I’m in one of our less interesting meeting rooms, but outside it looks very lovely.
– Well, it’s great to have you attending and speaking, and having a visitor from Toronto as well. And probably a good place to start is for you to talk a little bit about your background and the types of things that you do.
– Well, yeah, so right now I am a user experience lead at PagerDuty. Working on the products, to make some product design work, research, working closely with the product managers on strategy, working with data scientists on some of the more machine learning features. My background actually is in technical writing, so before I was a user experience designer, my amateur first job was in technical writing. And so I really love the ability to take something that’s really complicated and complex and kind of convert it into something that’s a little bit easier to understand, a little bit easier to use. From there I transitioned into service design, more on the enterprise level, understanding different ways that users and customers interact with enterprise products. Found that super fascinating, and then moved over to PagerDuty, where I’m actually working in the products design of enterprise products themselves, and working with very complex products. The users we deal with are software developers and operations teams, which often deal with very complex systems. Whenever they get us, any of their systems go down, PagerDuty will contact the right person to let them know what the problem is and help them coordinate a response to be able to fix that problem. So they’re often dealing with a lot of complex signals, and being able to show all that information in a kind of simple, streamlined, easy-to-use way is a fascinating problem to solve, especially when the emotions are running high, they’re very stressed, so it’s a very interesting design problem.
– Well is there anything that you’ve been working on or thinking about that you’re currently excited or passionate about?
– Yeah, so PagerDuty’s been making a lot of really deep investments into data science and understanding and doing research on some of the data that we have access to, that their users provide us with. So one of the things that I’m currently working on, just I’m kind of finished up a little bit of work on recently was looking at the way that users have set up some of their settings in PagerDuty related to their services, so all the kind of systems and infrastructures that they monitor. They give us a lot of information, mainly by entering it in, but then there also is a lot of information in the way that they interact with products. So what are they clicking on, how are they using some of the different features, and being able to work with data sciences to understand the patterns across both in the way that they set things up and the way that they interact with our features will help us understand what’s actually makes them more successful in terms of how are they actually being able to resolve for instance quicker, or being able to just be more successful with their product. So being able to understand that has been really interesting, and we’re trying to use that in a way that we can actually build some of the features into the product to be able to service recommendations and be able to give them the right information that they need to encourage those best practices.
– Well, that kinda leads us into the talk that you’re gonna give at the conference. The title is Building User Trust with Machine Learning Products, so tell us a little bit about what you’re gonna present.
– Yeah, so this is one of PagerDuty’s first investments in a machine learning feature, and the way that the feature works is software developers are receiving a lot of different signals from a lot of different systems. So whenever a system goes down, they’ll have a monitoring tool that maybe fires off a bunch of different messages saying the disk is filling, the disk is full. And often these messages can be very different, and there’s usually patterns involved in where the problem is contained, so this feature actually correlates those messages, groups them together so that they’re not receiving a hundred different notifications of errors, receiving one for the related problem. And one of the challenges that we’ve found is that while users, it actually does provide a lot of value, because the intelligence is able to correlate this this automatically for users. Often it was difficult for them to trust it, because there’s such high stress involved in these situations if they were to miss something. So users really were not confident in turning this on, because they felt more comfortable knowing that they had control if they were to write all these kind of manual rules and making decisions for themselves. But of course that’s a lot of work, it’s impossible to keep up with, it’s a full-time job in and of itself sometimes. And so part of the work that I did with the team was to design a feature that helped to build a little bit more transparency and explainability into this feature, to help people understand how it works, to help people be able to see how it works before they turn it on, and get more comfortable. And through that we kind of learned a lot about how people expect these automated features to work. People can sometimes misinterpret, because as a team we kind of understand it so deeply, but the users will have a very different interpretation on their end. And so understanding how they expect it to work, and being able to explain it in a way that helps them, and have a better mention model of how the actual algorithm works underneath. And so there was a lot of interesting lessons that we learned there, a lot of interesting ways that they tried to break the system. And in planning for it and trying to understand how people wanted to interact with it, there’s a lot of, I would say, twists on old, traditional ways of doing research, like user journey mapping, where converting it to how do you interact with a system that changes over time? How do you interact with a system where from one day to the next you might see totally different information? And being able to keep those things in mind, and then we’ll bring that into the research and design process, and also thinking about how to test and prototype that. How do you test and prototype something where the meaning of it really depends on the user’s unique data? So these are all some really interesting challenges, and I just wanted to share that story, and hope that other people might be able to take some lessons from that.
– Well, yeah, it’s certainly, I mean it’s a very exciting, emerging technology area, and I think a lot of people are gonna be very interested in your experiences, so they can relate to how that, what their own journey might be into working with that in the future. Well, thanks for taking the time to talk with me today, and we’ll look forward to seeing you when you make your trip over from Toronto in March.
– Awesome, thank you so much, looking forward to it.
– Right, thanks a lot. Bye bye, Claire.