Janine Zweig of the Urban Institute and Chad Sniffen of the National Sexual Violence Resource Center share how they worked with Uber Technologies to better understand sexual misconduct in the ridesharing industry. Janine and Chad share the process to define sexual assault, harassment, and other misconduct and then set up a system to track complaints. This new taxonomy and data collection will provide a more complete understanding of the problem of sexual misconduct, providing direction for prevention and response.
See the report on this project: https://www.urban.org/research/publication/examining-ubers-use-sexual-misconduct-and-violence-taxonomy-and-development-ubers-united-states-safety-report
More information is also available in this Urban blog post: https://www.urban.org/urban-wire/three-lessons-businesses-can-learn-ubers-collecting-and-reporting-sexual-assault-data
Transcript:
Susan [00:00:00] Welcome to “Tell Us About It: Victim Research Convos,” a podcast from the Center for Victim Research with support from the Office for Victims of Crime. On each episode of “Tell Us About It,” we talk to researchers and practitioners about their work, the tools being built for use in the field, and how we can work together to build an evidence base for victim services.
Susan [00:00:21] Today, we’re talking with Chad Sniffen, outreach manager at the National Sexual Violence Resource Center, and Janine Zweig, associate vice president for justice policy at the Urban Institute, about their work on the issue of sexual violence and harassment in the ridesharing industry. Welcome to the podcast.
Chad [00:00:40] Hello.
Susan [00:00:40] Can you introduce yourselves and tell us about your organization and your role in this project? Chad, let’s start with you.
Chad [00:00:47] Sure. So I’m actually representing two organizations in this podcast. So for the National Sexual Violence Resource Center, I’m the outreach manager and my function in that organization is to collaborate with our partners and to help facilitate our communications with other agencies, and I’m also primarily responsible for online content and making sure that our online content exists and is available. And so I am also representing Raliance, which is a partnership actually that the National Sexual Violence Resource Center participates in between NSVRC and CALCASA, the California Coalition Against Sexual Assault, and the National Alliance to End Sexual Violence. And so Raliance is a, basically a corporate consulting partnership where we provide consulting for for-profits and other organizations like sports entities and things like that around the topic of sexual violence. Most of our work with Uber and other corporate entities is done through alliance.
Susan [00:01:49] Great. Thank you, Janine?
Janine [00:01:51] I’m Janine Zweig from the Urban Institute in the Justice Policy Center, and the Urban Institute is a nonpartisan research organization based in D.C. We conduct policy research on a whole host of policy issues related to criminal justice policy and other areas as well.
Susan [00:02:10] Great. So tell us about this research project. What was the issue that you were researching and who all were your partners? Janine, I’ll start with you.
Janine [00:02:22] This project consisted of a collaboration initially with NSVRC and then Raliance with Urban Institute and Uber Technologies. The project was a very data-driven project where we were requested to help Uber manage their complaint and reporting data that they received through their platform, both from riders and drivers, about anything that might have happened on their platform related to sexual misconduct, assault, or harassment. So they wanted to understand how to best organize this data so that they can examine the data going forward related to the nature of the experiences that are happening on their platform and also be able to count the experiences and understand the prevalence rates related to what’s happening on the platform.
Susan [00:03:18] Chad, how did this research project come to be?
Chad [00:03:23] So, like a lot of interesting projects it came to be because the right person was really in the right room at the right time. So what happened was that several people who worked in our field, which is, you know, in general, the national organizations that provide technical assistance and advice and advocacy around these sexual violence issues were members of the Uber Safety Board, and they had been invited to become members of the Uber Safety Board. And in the process of the safety board working with Uber, they realized that the system that they had in place at the time for counting incidents of sexual violence was a system that if you had asked a software engineer to design a system for that purpose, that’s the system you got, right? So it’s not a terrible system, but it is not informed by experience with the topic, right? And so Uber realized that this was a data collection problem for them, that they weren’t really getting good data that was relevant to the issue. And so one of the board members for Uber recommended the National Sexual Violence Resource Center, so that was Ebony Tucker, who at the time, was with the National Institute on Sexual Violence. So Uber received that recommendation to talk to the National Sexual Violence Resource Center about how we could help them to develop a better system for collecting relevant data related sexual violence. And we, you know, are advocates and so the National Sexual Violence Resource Center excels at providing resources to advocates and providing information for lobbyists and for policymakers and, you know, speaking to the topic. But we’re not really researchers. So when this question came to us, it came to me specifically since I have a research background in my, like, other things I’ve done. But I knew that we needed a partner. And we had worked with the Urban Institute many times on various research projects related to sexual assault toolkits and other issues in sexual violence. So we knew that the Urban Institute would be a good partner for us in this. And so we reached out to them and that’s how it all started. And Janine and I have been working on this project for years now.
Susan [00:05:36] Janine, did this project involve actual data collection and analysis or was it devoted to creating the framework or the taxonomy for the data collection?
Janine [00:05:49] We did use some actual data that Uber received through their system on their application. But it involved really both avenues of what you just mentioned. We needed to go to the evidence base around measuring sexual assault to understand really what elements of this framework we want to make sure were present. And so we started there and then stress-tested the framework through several rounds of receiving what they call tickets, where we were able to examine actual user input and then examine whether the framework was appropriate, and then did iterations of the framework based on that. So some of the elements we really wanted to make sure were present in this, based on the research phase on measuring sexual violence, is that we were using behaviorally specific questions. In other words, instead of asking, “Were you raped?” the field has learned that it’s easier to ask, you know, things like, “Did you have sex when you did not want to because you were held down, because you weren’t allowed to leave, because someone used violence against you?” Things like that versus asking, “Were you raped?” because that’s a very subjective definition. People don’t interpret the word “rape” the same way, but people are interpreting behaviors against their bodies — did the person touch your leg? Did they ask for a kiss on your mouth? — those kinds of questions the same way. So we wanted to make sure the framework was very behaviorally focused so that the users of the taxonomy are bringing their own subjective thought process and interpreting what they’re reading in a complaint and trying to use words that might be interpreted differently across different users. We wanted it to be as objective as possible and have it be about behaviors rather than other kinds of words. And then we went on to define certain behaviors — certain words as certain behaviors, and so “hitting on me” is flirting, is a flirtatious behavior, that kind of extra step of defining things to make sure that people were applying the taxonomy categories the same way. So that was — the first part was to assess the literature and see where we wanted to take the framework. And then once we had the framework in place and a series of categories that we wanted to make sure were mutually exclusive, so a single complaint wouldn’t go across multiple categories, and then what we called collectively exhaustive, meaning any complaint that was received could fit somewhere in our taxonomy. So it was two defining features as well, guiding the work. We came up with a categorization system that ultimately was 21 categories. It was a little bit longer to start, but through the stress-testing and through the collaboration with Uber to understand what’s actually feasible in a business environment rather than just social science environment, we were able to pare that down to have the categories be a series of sexual misconduct categories and a series of sexual assault categories.
Susan [00:08:55] Great, that does sound really involved. Chad, how long did it take you all to get this project off the ground and then to complete? Because it sounds like a very involved process.
Chad [00:09:08] Wow, it was. I think that really depends on what you consider complete. So we thought it would take a few months to develop the taxonomy review data [and] stress test it. It took about a year actually in practice. And that was due to learning how to work, you know, with a for-profit like Uber, um Uber, learning how to work with us, learning to basically trust each other, you know, that we all have the best intentions, you know? And also learning — we didn’t understand at the time what kind of data we needed to look at. So we had to look at data, a lot of data, before we realized we needed a different kind of data or we weren’t seeing enough of certain types of data to look at, right? Because sexual misconduct and sexual violence on the Uber platform is a tiny, tiny, tiny percentage of all the complaints they receive. It’s a vanishingly small percentage. But Uber is a massive platform. They have 15 billion rides a year. So a tiny percentage of 15 billion is a lot of, you know, sexual misconduct and sexual violence. But it is a needle in a huge haystack. Even though it’s a gigantic needle, the haystack is even bigger. And so we had to figure out how to — which data were the right ones to look at. How do we even find those data? You know, where are we missing data because of the previous system? That was an issue — we needed to understand the previous system well enough to understand where missing data might be, right? And even actually — so later on, Uber had another company doing evaluation work of the whole process for them, and they even recommended different machine learning models that we had not thought of to find data that Uber had not correctly identified in the past or identified as being relevant in the past. And so, yes, it was a very involved process, but a lot of it was relationship-building to be honest, a lot of it was figuring out what we need, what they need, how we can communicate effectively and also, at the time, Uber was going through a lot. Uber had had a CEO change. They had a lot of publicity on this topic that really made everything feel a little cautious, you know, because they wanted to respond to the concern, but they didn’t want to make it look like they were kind of farming out the problem that they were owning the problem, right? And so, and we also, as you know, as advocacy organizations, we have our own reputations to think about, right? So we have to make sure that we are not turning into their mouthpieces, right, that we are, you know, maintaining the integrity of our organizations and that we are doing work that is for the betterment of the topic and of survivors of sexual violence and and they appreciated that. But that also meant that sometimes we had to do some negotiation. And so not because we had different goals, but just because we had different concerns, underlying concerns. And so that just required a lot of time and relationship building to accomplish.
Susan [00:12:24] Yeah, that all makes sense. Janine, from a research perspective, what were some of the challenges that you saw?
Janine [00:12:31] So I would say very similar to what Chad talked about around trust building, but for different reasons. His was referring to the advocacy piece, and ours referring to, you know, the way we work at Urban is try to be as objective as possible and let the data drive the story that we’re looking to tell. And so the objectivity piece was another element of making sure we had enough trust among our partners at Uber and with NSVRC to be clear where we would be drawing the line and what we would say about various pieces of the research process and then the data analysis. And then their, ultimately their reporting that came out in late 2019. And so it was a long process of trust building and discussion around these issues and productive as well, very productive to be a really, I think, effective collaboration among the three groups.
Susan [00:13:31] So, Janine, what is the next step for this area of data analysis and research?
Janine [00:13:39] So Uber’s vision for this taxonomy and the way that they had us create it as a open source white paper so that other industries and other businesses within the transportation industry would take it up is that they were hoping other places would use this. And so we are in the process of continuing to look for transportation agencies and other organizations in the hospitality world to consider taking this up and implement it in their own work for their own users of their platforms. And as a part of that process, we’re learning where our guidance in the white paper and how the taxonomy can be refined. And so, again, Uber’s vision was if it needed refinements down the road, we’d be able to do refinements down the road. And so what I think our next steps are, is to identify the areas where we want to refine pieces of the taxonomy. And when I say that, I’m going to use the old analogy of apples to oranges. So, you know, researchers often say we want to make sure the data that we’re comparing are the same, so we’re not comparing apples to oranges. In this case, what we’re trying to do is put refinements in to make sure we’re — that all the apples are the same variety. So instead of comparing Macintosh to Delicious, we’re looking at the same variety of apple for every category. So it — it’s tweaks around the edges to make sure all the apples are the same. And then we do hope down the road further that as more places implement the taxonomy — and so we know about the nature of what’s happening on these platforms and we’re able to count the extent to which they’re happening — we’d also like to understand how they’re linking this categorization process to their response strategies, and then and then also prevention strategies beyond that and whether or not the taxonomy is effective at helping them be responsive to what’s happening. And [inaudable] evaluation research, which hasn’t happened yet in terms of how effective it is at helping with those processes.
Chad [00:15:47] Right, and if I could add to that, you know, another area of work that I think will stretch into the future is, is the usability piece of it. And so the taxonomy that is actually published is not actually the initial taxonomy that we had developed. So we had developed an exhaustive inventory of different, you know, incidents of misconduct and assault. And it was super unusable at scale, right? It was just, it was just too detailed. It required — there was just too much nuance in it. And so we worked with Uber to make it more usable at scale, especially with people who are seeing a decent amount of training but are not specialists in this field. And so, and so — but in the process of making it usable, we also took out detail, right? So there needs to be — I think we’re still finding the perfect balance point between usability and detail. And also, at the end of the day, what does what does the information do for you? Right? That’s a question like, so what can you — does it really matter if you know the difference between one specific type of assault and another if you respond to them in the same way. So you know or can you find differences that lead to different responses and different outcomes. And so I think that’s something we’re still learning about as other as multiple companies use the taxonomy and talk to us about it. And then we have to figure out a way to make those changes, but not make them precociously. You know, we have to make those changes in a way that can ideally work for everybody because, you know, we envision that the taxonomy should be useful in all circumstances. It’s meant to be a generic taxonomy. It’s not meant to be specifically for the ridesharing experience, although it was developed with that. And so in some ways, it is geared towards that. And so there are also ultimately we would like this to be a general purpose tool that can be used for anyone trying to develop a measure of what bad things are happening in a certain place.
Susan [00:17:59] So that’s great. I think a lot of folks, me included, don’t always think about research in in terms of how important the process is to figure out exactly what you are measuring and how best to measure it and how significant it is to go through that process so that your end result really means something. I think a lot of us end users tend to go right to these “So what’s the number? You know, how many?” without realizing that you all, by going through this process, are not only setting up to make sure that the numbers will matter, but you’re educating your partners, you’re contributing to the understanding of the nature of sexual violence within this business setting. And I just find this fascinating. Let’s talk about the relationship. You are both spoke about the need to build trust among your organizations and Uber as you were going through this process. But let’s take a step back. Janine, I believe you and Urban have worked with NSVRC before. What is it that you would credit your strong collaborative relationship to?
Janine [00:19:23] That’s a great question. I think that it’s been with NSVRC, I would say inherently wanting — coming into the relationship with inherent trust. So we really started in the earlier days with bringing NSVRC into studies where we wanted to make sure an advocacy voice, that an organization could elevate survivor voices as we’re putting our research design together. As we were writing up our research findings, helping us to interpret the findings in a way that was both useful to practitioners and honored survivors voice to the extent that NSVRC can represent that and do represent that. Even though we would build in things like focus groups to talk with survivors on these studies, it was critical to have NSVRC’s voice and advisement on these projects. And it started very early on, way back when the Prison Rape Elimination Act was first passed in ’03. The first project we did on that — Pennsylvania Coalition Against Rape joined us on that project, and later on NSVRC to bring together those two voices, the researcher perspective, the practitioner perspective, and the extent to which we could elevate survivors’ voices. And then that proceeded to be, over the years as other projects came up in the sexual assault world, we routinely would bring NSVRC in for advisement. And so when they reached out to us on this Uber work, it was a fascinating opportunity to actually do the work in a much more collaborative hand in glove way than just as advisors to one another. And we really have worked as a seamless team with Urban staff and NSVRC staff co-conducting the research, co-reading the tickets, classifying the tickets, because one of the critical elements was to be able to demonstrate this taxonomy can be applied the same way regardless, regardless of who’s applying it, as long as we have the same training. So within Uber, the auditors are applying this the same way. And then across organizations. And so our internal team, meaning the staff from Urban and NSVRC who were on the project together, did those same kinds of activities together. We would, you know, separately be taking categorizing our tickets and then coming together to see where we had alignment and discussing the challenges around those things. And so it was it was a very collaborative process where we were working on the same tests together in a way that brought our organizations together in a little bit different way than in past roles.
Susan [00:22:05] So, Chad, from your perspective, as the practitioner end of this, what would you say about the relationship and the keys to your success?
Chad [00:22:14] I would definitely agree with everything Janine has said in that, you know, we had an established history. So it wasn’t just me that decided to talk to Urban Institute, it was, you know, our CEO, who was the director of NSVRC at the time. She’s a CEO now. And I said, hey, well, we worked with Urban successfully in the past, let’s talk to them first, right, and see what they think. And then if they don’t think this is a fit for them, we have other people we can talk to. But they were, you know, the first group that we approached. And so that, that existing relationship and past work with each other really helped. Also as individuals on the team — so the team at the time was myself and Janine, and a woman named Julia and a woman named Sally Laskey. We each came to that project with kind of both advocacy and research experience and so that helped a lot. So Janine has been an advocate. Sally and I have both done research in graduate school. My, actually, my grad school research was specifically around something like the taxonomy working for Dr. Mary Koss, who developed Sexual Experiences Scale. And so it helped a lot that I had relevant experience and that Janine had relevant experience so that when we were — I could understand what they needed from us, and they could understand what we needed from them because we had enough similar background that we didn’t have to do a lot of explanation and a lot of kind of build up a pretext, right? And so that was very helpful. That’s of course, that’s not something you can bake into every pie, but, you know, it did help make this relationship work a lot more smoothly.
Susan [00:23:54] Thank you, that does explain why you work so well together. Now let’s go back to this type of work. After your work with Uber and you know, this has been a long process and very iterative. What would you say are the lessons for other businesses or entities that would want to measure sexual violence? You did say that one of the purposes of creating this was so that it could be shared, that that was one of the ideas from the get go. What would, what would you advise other businesses or what are the lessons you’d share?
Chad [00:24:29] So in terms of a taxonomy we’ve learned so far from businesses is that this is actually a resource intensive project for multiple reasons. One, that data have to be — treated in a very special way, not only because these are on safety issues, but also, you know, the taxonomy itself is designed to avoid bias around the topic. That’s why behaviorly specific questions are important. And so but that, but the taxonomy itself does not include, does not guarantee that there won’t be bias in the way that complaints are processed. So there needs to be a lot of training of the people who actually work with this topic. There needs to be somewhat, you know, intense data collection and dataset sanitization processes. Nothing sanitization to exclude data, but sanitization to make sure that you’re getting the right data that you need, right, and that you’re not. And actually, what we found is that — what we thought we had found is that at some point you have to come to a decision of are you going to over-include or under-include? Because this is a very difficult process to talk about. And it’s a very difficult experience to talk about. And people communicate about this in in a variety of ways that are not always exactly what you need to hear as a person working with data. And so you have to at some point as a business, make a decision, are we going to overcount or undercount? And is the system we’re going to develop going to be one that tends towards overcounting or undercounting, not undercounting because you’re trying to minimize the problem, but just undercounting because you’re only getting, you’re only accepting the really clear circumstances that are expressed in exactly the right way as data points and not the nebulous circumstances. And then which one is safer? And so what we found that for large companies that they’ve actually been willing to overcount because they recognize that this is a safety issue of critical concern and that they served their customer base, you know, the people using their platforms better by overcounting than undercounting. They take more of a media hit, but they’re taking more media hit anyway, right, so it doesn’t it doesn’t help them to undercount. Which is, you know, frankly, a dynamic that a lot of universities are still trying. We were very pleased to find that that’s something that we found with businesses and especially Uber.
Susan [00:27:01] Janine, anything to add to that? About the lessons learned for other businesses or other entities?
Janine [00:27:09] I think that from my perspective, as Chad said, having been an advocate way long ago and then a researcher now for many years. Businesses have a real opportunity. The scope of reach they have is so much further than so many of the programs we’ve studied in the past or so many of the advocacy efforts that are locally based, which are critical and I’m not saying anything that would undermine those in any way. Those are critical elements to our victims’ service structure that are needed. But a company like Uber, and the scope and the reach they have to be addressing these issues, it feels imperative that they do so. And so from my perspective, having come into this process a little bit skeptical, skeptical about what their goals were and what did they really want to get out of this? I quickly realized the people I was working with day-to-day were very authentically interested in preventing sexual violence on their platform, and that by doing so, they would be preventing a vast amount of sexual violence. So for other businesses, I would say I encourage them to do this because for the greater good, the scope of their reach to prevent sexual violence is enormous.
Susan [00:28:28] Well, I want to thank both of you for your time today. This has really been so encouraging to see research and advocacy and business all coming together to take on a big problem in a way that’s really going to make a difference. So thank you both for sharing your experience with us today.
Susan [00:28:51] We hope you enjoyed this episode of “Tell Us About It.” If there are research or practice experts you’d like us to interview or research tools you’d like us to feature on this podcast, email us at podcasts@victimresearch.org.
Closing [00:29:05] “Tell Us About It” is a production of the Center for Victim Research funded by the Office for Victims of Crime’s Vision 21 Initiative through Cooperative Agreement Number 2016-XVGX-K006. The Office for Victims of Crime is part of the U.S. Department of Justice’s Office of Justice Programs. However, the points of view and opinions discussed on this podcast are those of the host and expert contributors and do not necessarily represent the official position or policies of the U.S. Department of Justice.
