November 29, 2021
Evolving Traditional Product Metrics to Grow Positive Social Impact
Assessing a product’s impact is complicated. Here's how Exygy approaches creating product metrics to support meaningful social impact.
.png)
Our goals for today are to plant the foundation of accessibility, cost, and time savings for our digital products. We're going to grow our approach to metrics that support your organization's mission and guiding principles for social impact. And then we're going to create an objectives focused product roadmap that is supported by mission aligned metrics to help people thrive.
Our mission is to partner with social impact organizations to design and build technology that improves lives. And so part of what we need to do is figure out: are we actually really improving lives through the technology that we design and build? And that's really been our journey over the last several years. Our focus areas include affordable housing, health care, equity, and social safety nets. We've gotten an opportunity to work with an incredible group of organizations from civic organizations like the City of San Francisco and the City of Oakland, to an amazing project with CARE International in Tanzania.
As product managers who work even for Exygy, it feels like our industry's having a little bit of a crisis. Like some of the ways we've measured success in our products haven't necessarily generated healthy outcomes for individuals.
When we see examples of this, it's easy to put ourselves in an “us versus them” thing. While we see Mark Zuckerberg on TV a lot, there is also some acceptance of the role that we play. That's where we need to think about the technology that we design and build impacts individuals. We have to be accountable for it, as well. It's time to evolve our metrics from focusing only on the growth of the organization, to be outwardly focused on supporting the people that are using the technology that we built for them.
Stepping back, let’s explore a little bit of a history of product management. Once technology got to a place where we could think about how people were clicking through websites, we had the birth of user experience design. From that, we get into the search engine wars, and Google seems to have come out on top. And I think one of the reasons why they came out on top was Google Analytics. From this combination of really getting better user experience, and combining that with data, we really have the emergence of product management within design and development of technology around 10 or so years ago. We have this framework called “AARRR metrics,” made popular by Dave McClurer. These metrics are how we started quantifying how successful the features we were building were for the organization. But there's a gap between AARRR metrics and today, and that's what we're going to talk about.
One of the challenges within our space of social impact is there's a bit of a resistance to thinking about the technology that we're designing and building as products. And we really need to think about it in terms of the product. There is a lifecycle for every piece of software out there. It starts with the inception. There is a kind of an introduction and figuring out like, “who are my users?”, “what kind of needs are we satisfying?” Once we hit product-market fit, we can go into a growth space, and then ultimately maturity and decline. This applies to all software — whether it be a SAAS application or a website — they all go through this lifecycle.
Let’s take a look at the AARRR metrics. Remember, there's nothing inherently wrong with these; acquisition of users is good, activation of users is good — especially for for-profit organizations like Exygy. There's still a place for these metrics. But notice something about them: they're really about benefiting the organization, right? They're benefiting the people who make the technology, not necessarily accounting for the people who are using the technology. So how do we know that we're making the impact that we want to make with our technology that we design and build? Because if we think about impact, we might actually be creating the wrong kind of impact, and we want to be making a positive impact, right?
I live in Berkeley, California. It's a lovely place to live. I moved here in about 2010 and I started a garden in the backyard. It was always my dream. I used to live in New York City. When I started, I knew nothing about what I was doing. And I have this 4x8ft foot garden box. I threw in some tomatoes and basil and corn — but you can't really grow corn in Berkeley! It's not really the right climate for it, and everything's kinda crowded there. Nevertheless, I started putting plants in the ground and learning from that.
And a great place to really begin learning about creating digital products for people to thrive is really starting with what I call “ACT metrics:”
Accessibility isn't really a new concept for everybody. It's. I think Jeffrey Zeldman has been talking about web standards for 10 or 15 years, but the reality is we're still in a place where we're not doing it very well. This ITIF study found that 92% of the federal government's most popular sites, didn't meet the basic standards for security speed, mobile friendliness, and accessibility. So even though we've been talking about it for a long time, we're still not doing it very well. Let me show you some examples of how we can do a little bit better.
The example I wanted to show here is an affordable housing platform that we created for the City of San Francisco, called DAHLIA. Now we’re adapting it into a software called Bloom Housing. On the slide above, we have a browser plugin called WAVE that is free to use. It will scan a page on your website or your web app to show you where those accessibility problems are. It'll highlight the content that is not structured well for screen readers. It's going to highlight errors in color contrast, which is a really big one. Even people don't really realize how pervasive vision impairment is.
Moving on to an example that we use quite a bit in the work that we do: the US Web Design System (USWDS). This was created in collaboration with some of the organizations within the US government, along with 18F. Their mission is to make it easier to build these accessible websites for the government. And they've established a mission, and then a certain set of design principles that really are focused on end user needs. So starting with the user needs and building their trust, embracing accessibility. The way that works out in practice is that folks who are building new websites for the US government can use this front end framework, this design system, in order to create consistency from when folks are going from one site, one site to the next. This we've used for some of the work that we do for San Francisco, as well as some of the work that we do for the City of Oakland.
So when we think about cost savings, we’re thinking about these three areas around hosting software licensing and software engineers. Hosting is something that has to be considered pretty early on when we're designing and building new technology or evaluating auditing once that technology is in the wild. Hosting sites can range from a few dollars and then not be necessarily super robust and secure, to thousands of dollars a month and offer more secure and robust solutions. So really factoring in those costs from the beginning is super, super important. The software licenses here it's like, what is that -- build versus buy? One argument is the license to use proprietary software could actually be more costly than if you were to build certain things from scratch.
If you're building data visualization dashboards, like Tableau, we've looked at it and it's great software, but it is very expensive. So I'm really kind of digging in and evaluating that as important from a software engineering perspective: there is the average cost of what it costs to build these things, but also the technology itself comes into play. You may factor in the availability of software engineers in a particular area. I mentioned that work that we did for CARE International in Tanzania. Internally, Exygy is standardized on sort of a JavaScript technology stack that we use for our affordable housing platform. In Tanzania, finding engineers that could support that software long-term was very difficult. So we really honed in on agencies in Tanzania that could support it with the technology that was popular in that area, which happened to be a PHP framework. We ended up building out this dashboard in PHP.
This example I have on the right-hand side is work we did with the San Francisco Unified School District. And for them, it was a process of unifying all of their web systems under a platform that could support both the main website, as well as these individual school sites. When we started working with them, they were using a proprietary platform for hosting school websites that was really costly. And so that became a real metric of success for them: to save money from the software licensing costs and from time savings perspective. We think of it with different perspectives.
Consider ease of use for end users or people who are actually using the software: how long they're taking to complete either the service in this case, or the thing that they're trying to do, the job that needs to be done. The ease of actually maintaining the site or editing content often gets overlooked. And these are the administrators that are keeping the content up to date. With our affordable housing platform for example, it's actually the folks who have to build listings for that platform.
To illustrate the reduction of calls or in-person visits, I really liked this example on the right hand side of the slide above. This is work that the Judiciary Council of California has done, and they've been real trailblazers in the area of self-represented litigation in the court systems -- meaning folks who can manage their case online and not have to go in person. And that's a huge time savings for people. They've created a really wonderful step-by-step process in their self-help guide that provides services for issues like like name changes, marriage licenses, divorce, gender identity changes.
So that's all to say with accessibility and cost savings, we must meet these basic needs of accessibility, cost savings, and time savings before we can expect to make meaningful, positive social impact. That really has to be the foundation.
After we've established that and built some of our soil, we're ready to grow an approach to metrics that support our organization's mission, guiding principles for social impact. So come on back into my garden, and it's about 2016. So I've been doing this for about five years at this point, and I've kind of broken free of that four by eight bed.
I'm starting to build the soil. And from that, I can kind of give up control of where those plants are going to be and get them in a place where they're really starting to thrive. So I've expanded out the beds. I've added flowers to the mix. I didn't know this before, but when you add flowers, it brings in all these pollinators and your yields just really increase.
Bringing it back into our product metrics, what we want to do is sort of refocus our metrics by aligning them on our mission and principles for social impact. So there's still a place for those AARRR metrics, but they just need to be counterbalanced by those in our mission and principles. We don't need to be afraid of using them. And so how do we get started with this? First we need to align our product goals to the mission and guiding principles.
And this is kind of a made up example, sort of loosely based on a civic organization. But let's say the civic organizations mission is designed to build technology that improves lives. The principles within this organization are focused on equity, simplicity, and trust. Say we’re starting to build an application. To start, a lot of stakeholders will come to the table and bring a lot of goals: they want the site to be more readable, they want people to find the information, they want it to be transparent, mobile, responsive and certain things that maybe fit outside. They want a department org chart listed, or they want a Slack channel built. When we start aligning these goals with those principles of social impact, then we can start to see which ones don't really fit very well. So Slack channel -- if we don't really know how it contributes to equity or simplicity or trust, then maybe we have this way of just kind of putting it aside. When we're starting to build our design and build technology, we're also collecting all these features. So we're starting to build out this backlog of features.
So our product goals should not be about control. Instead, they must connect to an existing habitat or community and contribute to it in a way that directly supports the well-being of individuals. Once we think about that, we are getting in a place where things can really start blooming, and from that, we can create an objectives focused product roadmap that is supported by mission aligned metrics that help people thrive.
It's now 2021 and things are really starting to thrive in my garden. I'm understanding how the light changes throughout the seasons. I've dialed in the type of veggies that work really well in my climate. I'm seeing all these native bees and pollinators and they have a beneficial impact. So there's really starting to get a habitat for things to thrive in my backyard.
Now, again, bringing it back into our product management techniques. We've adapted a series of techniques from folks that are, pretty well-known in the industry. And we've kind of just adapted them in a way that works really well for us.
We love doing this product vision board exercise created by Roman Pichler at the kickoff of new projects. We also love redoing it on a quarterly release cycle, whatever cadence works best for you to reevaluate it. What this does is it brings the stakeholders to the table with their different perspectives. IDEO has the concept of sort of diverging and converging at this stage. We're really wanting to be in this sort of divergent area where we're just sort of collecting all the ideas.
So with this exercise, we make sure we understand that everybody has the same big vision for the product. Then also making sure that product vision supports our organizational goals. We're getting a sense of our target group of users and their needs. These are sometimes called “jobs to be done.” And then in this product column we really just want to collect all these things that people want in the product, like what they think the product should do. These are broad features and strategy and any sort of relevant business goals or context. For example, are there some revenue goals that we should understand, or other sort of business context areas that we should account for at this stage?
So there's a lot of stuff that comes out of this. It's going to sort of diverge, it's just an example that came out of work that we did with the [inaudible] foundation. And there's kind of a mixture again, we're not, we're not, we've maybe brought user research to the table, which is also always good to do beforehand. But at this stage we might not know exactly for right on target with the groups of users. We'll know more and more as we get towards that product-market fit area. We have needs outlined here product and business goals. And so it just looks like this big collection of information.
And so how do we get from here to a place where we can really think about metrics that are meaningful? We want to converge back on those principles we thought about earlier. So yeah, so all of those things that came from this maybe product column, if they're features, maybe we need to think about moving them to our backlog. If they're not really objectives that fit into these principles, then maybe they need to be pushed aside or refactored. But it is a good exercise to make sure everybody is being heard.
I'm sure a lot of people are familiar with the concept of OKR objectives and key results within the organizational level. But I actually really liked them almost more when we think about applying them to products and product metrics. And the reason being is they have a natural structure of the objective being really aspirational. And I think that's a lot about the work that we do in social impact -- we are really being aspirational. It gets us out of bed every morning. If we tie these objectives back to those principles and apply them to the product that we're designing and building, we can look at them as: our aspirational objective is to make the website that promotes tech equity. The website is simple to use and the website promotes trust in government, and we can create our set of key results that directly support those objectives. So for tech equity, all of our websites on all of our pages on our website are compatible with mobile devices. We reduce the average load speed by 20%. So you can see how this becomes kind of a powerful way to set your objectives and the way you measure success based on those objectives.
So now how do we bring that into an actionable product roadmap? And again, kind of leaning back on this technique that Roman Pichler introduced. This goal oriented roadmap is a way to kind of break free of some of the approaches to product roadmapping that you might find in like JIRA or Asana where they're really just timelines. They're not really like agile roadmaps where we think about epics and doing sort of parts of epics in this time, and then doing another part of the epic, another part of the epic.
This grid view of your roadmap kind of puts aside the timeline for now. So we have the series of epics or our epics for our product that we're building. Those become sort of the names, the goals, the dates, when we want, when we are targeting them to be released, the goal that they're supporting. Now we can start connecting this to our product backlog through the features. And then each metric is aligned to the goal and the features that are actually going to achieve those metrics. And I'm just going to blow out like one epic for us to take a look at how that, that, that looks like.
So the epic is a civic service redesign, and we're targeting to release this in Q4 2021. Our objective that we're supporting is a website that is simple to use. And then we have our series of features that support that. And then the key results are our metrics. So key results are how we know that these features have done what we've set out to achieve.
The other part of this that really constantly needs to be reevaluated is that we need to get continuous feedback from users. We always need to be ideating and prototyping user testing, so that we get that feedback before we go into development. In doing these cycles, we can constantly check ourselves on answering the question: are we actually meeting the needs of the users? Are we actually setting the right metrics that we need to do?
It's not easy to truly understand whether the technology we create is doing good or harming a community. To keep your habitat thriving...
Back to my garden. I've got some native lupine here. You can't really see them, but there's a lot of little tiny little native bees kind of zooming around in there. A thriving habitat.
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
