In this episode of think civic, Evan and Ajay spoke with Sha Hwang, co-founder and COO of the Nava Public Benefit Corporation. Nava is a public benefit corporation working to improve critical government services for vulnerable populations. In this episode, we had the opportunity to discuss:
Nava’s formation and the founders’ work around Healthcare.gov
How Nava works to improve access to public benefits at all levels of government
How Nava builds teams to work on projects and ensures their work has a lasting impact on the government organizations they work with
Nava’s principles when building teams that work on government projects
Where the future lies for tech in the public benefits space
Episode credits: Hosted by Evan and Ajay
Cover art created by modifying Abstract Shapes © Nathan Covert (Licensed under CC BY 4.0)
References mentioned in the episode:
Case study on the billion user load test we ran on Nava’s Scalable Login System
Our approach to helping CMS optimize its cloud platform
Designing a more personal healthcare experience
Nava PBC on Twitter
Sha on Twitter
Nava is hiring! Our careers page.
[00:00:00] Evan DeBroux: Hello and welcome to the Think Civic Podcast. I'm Evan DeBroux and I'm delighted to be joined today by Ajay Jain.
[00:00:25] Ajay Jain: Hello y'all, great to be here.
[00:00:28] Evan DeBroux: So, yeah, you may be wondering, like, man, your RPM has really dropped off a cliff since July, and I, I would tell you, uh, yeah. Yeah, it has. Definitely, definitely. Yeah. We, we just can't really escape the responsibilities that in our lives. I mean, we've been super busy, uh, and very busy bees. Yes, Yes, yes.
[00:00:50] Ajay Jain: Uh, Buzz, Buzz, buzz.
[00:00:52] Evan DeBroux: So, yeah, like I, I'm wrapping up, uh, coaching a soccer season. Right now I'm actually dressed for our final regular season game, which I'll be going to in about, uh, 45 minutes.
[00:01:02] Uh, and now I'm prepping to coach basketball and, uh, yeah, it's kind of crazy to think that that season's three and a half weeks away. Uh, but yeah, it'll, it'll be fun. Um, also, we've been finishing up our family's move, or trying to finish up our family's move to our house in Sister Bay. Uh, you know, in my imagination, uh, the act of physically moving would've been the easy part. Or no, not really easy part, the more difficult part, but really it's been just trying to get established. Like, I don't know, like who my doctor is. I don't know who my dentist is. I haven't had a physical in three years. It's kind of kind of crazy. And then like try to buy a car.
[00:01:40] Ajay Jain: And you might wanna get a physical sometimes too.
[00:01:42] Evan DeBroux: Yeah, yeah. You know, probably wouldn't be the worst thing. Right. I mean, Covid threw a big old wrench into that.
[00:01:48] Ajay Jain: Really did. It really did.
[00:01:50] Evan DeBroux: Yeah, I kind of need to get on that .
[00:01:53] Ajay Jain: Well, speaking of moving, I, I've actually been in a very similar boat, Evan. Um, I know ironically when we first started this podcast, it was kind of towards the end of the pandemic, but still at the point when we were still, you know, doing things extremely virtually and still quarantining, um, in the early parts of 2021.
[00:02:10] And so at the time I was living in the Chicago suburbs and then prior to, um, Our starting our second season, um, I had actually moved to Austin, Texas and, uh, did a lot of living there. Um, especially over the last few months. Did not travel at all and just kind of stayed in Austin and enjoyed the summer.
[00:02:27] But again, on the topic of moving, even though I only moved to Austin relatively recently, I actually am moving to Washington, DC uh, in. So just about two months, which I am really, really excited about and being able to be, um, in a city that I used to live in, albeit only for three months compared to the rest of my life, living in the central time zone, whether it's been Illinois or in Texas. Um, I'm just really excited about the whole vibe of the city and being in a place that's very familiar to me and close to my family, and a place where in Austin, Texas, it is the middle of October and 95 degrees in fall, and I should be drinking a nice hot apple cider and wearing a sweater and looking at the leaves fall, and I am massively looking forward to doing that and wearing warmer clothes and layers.
[00:03:12] In Washington DC So that is something that has been going on in my life, uh, over the last little bit of time.
[00:03:18] Evan DeBroux: Ooh, man, nice warm, uh, cup of apple cider. I'd go for that right now. I think I'm actually gonna have some of that this weekend.
[00:03:25] Ajay Jain: It's 90 degrees here, and I'm still tempted by having some apple cider right by now.
[00:03:28] Evan DeBroux: I just hope your winter's mild because I'm sure I'm expecting like 90 inches of snow where I'm living. So, you know.
[00:03:35] Ajay Jain: So when I lived in DC. Uh, my second semester of junior year, so I had actually moved there in January. And one of the things that I learned about living in DC at that time, uh, was besides the part that DC is just an absolutely incredible city and I really enjoy the energy of the city, is that it will snow occasionally.
[00:03:51] And if there's a little bit of like sprinkles and flurries, uh, it's really beautiful looking out like your big apartment windows and seeing all that happen. Uh, but if you get a little too much snow, like an inch, the city does shut down and you would think like being in that part of the northeast. It does snow there at least once a year.
[00:04:07] But you know, uh, it does shut down. So I'm hoping that we don't get your levels of snow. But I won't complain about a little, little tiny little bit of snow. Especially cuz if you get that in Austin, well, they're done for a week.
[00:04:20] Evan DeBroux: Yeah. Doesn't the energy, uh, grid go down completely? I mean, we... uh, is that too soon?
[00:04:25] Ajay Jain: We were a little, we were a little concerned.
[00:04:26] It's, uh, back in February. We had a tiny bit of a winter storm and there was like a quarter inch of ice, but the entire city shut down for like an entire day. And I remember like I was hosting a party and I had to like cancel my entire plans just cuz like it was in the thirties. And the moment where it goes below freezing in Austin, people start freaking out.
[00:04:45] Like, do we have to trip the faucets? Do we have to do this? Do we have to do that? And for me, this was actually my first and only winter where I was in Austin, Texas. And all my previous winters had been Chicago, Illinois. And I was like, people do what? You drip what? Isn't it just supposed to be naturally okay.
[00:05:03] Like what, what's going on? Why is everyone freaking out? But, um, but I get it now and I'm glad that I don't have to do that freak out again anytime soon cuz I was kind of partially with one of them when I realized the infrastructure actually could not support cold weather. Um, but glad to be in a place that not only allows the supporting of some cold weather, but where my walking closet can now also have more cold weather clothes. So it'll be really great.
[00:05:26] Evan DeBroux: And I'm glad that the move's gone so well for you. And you know what? We are also glad to be back.
[00:05:32] Ajay Jain: Uh, yes, Yes, we are.
[00:05:34] Evan DeBroux: Swing this back on topic of bantering on about snow. Uh, anyway, Yes, in this episode, who did we talk to today? Well, we got to talk to, uh, Sha Hwang.
[00:05:45] Uh, Sha is the co-founder and current COO of the Nava Public Benefit Corporation. Uh, for our listeners who may not be aware of Nava's work, uh, Nava is a public benefit corporation working to improve critical government services for vulnerable populations. Uh, we had the chance to discuss, uh, Nova's formation and the founders work around healthcare.gov. Uh, if you like, load testing, uh, lengthy segment on that.
[00:06:11] Uh, Nav... how Nava works to improve access to public benefits at all levels of government. How NAV builds teams to work on projects and ensure their work has a lasting impact on the government organizations they work with; and where the future lies, uh, for tech in the public benefit space.
[00:06:30] Uh, so anyway, without further delay, uh, this is our interview with Sha.
[00:06:42] Ajay Jain: All right, well, welcome to the podcast.
[00:06:44] Sha Hwang: Thank you for having me.
[00:06:46] Evan DeBroux: Well Sha, just to start off, can you please tell us who you are and what your role is or your roles are at Nava PBC?
[00:06:56] Sha Hwang: Yeah, absolutely. So my name is Sha Hwang. I'm the COO here at Nava. I'm one of the co-founders. I use he/him pronouns. I'm based in Brooklyn in New York.
[00:07:06] Yeah. And have been working with Nava since the beginning.
[00:07:09] Evan DeBroux: All right. So just kind keeping that rolling here. So as one of the founders of Nava, we're kind of interested in knowing how did Nava first get started and what are some of the impactful projects the organization initially worked on at its nebulous?
[00:07:27] Sha Hwang: Yeah, Nava, we're a public benefit corporation and we got our start when myself and the other founding members of Nava were pulled in to support the healthcare. gov tech surge during that first open enrollment in 2013-2014. Since then Nava has grown. We've been focused on working to make sure that government services like healthcare.gov are simple, effective, and accessible to all.
[00:07:57] And we've been doing that now, uh, for now, Nava's been just across seven years, we've been working across the federal, state, and local levels as well.
[00:08:06] Ajay Jain: Sha, so through our year and a half of hosting think civic, Evan and I have interviewed a multitude of guests who have worked on civic technology and government problems, as like you just said, at the federal, state and local levels.
[00:08:18] And in many of these situations, these individuals or organizations are often mainly working with one level of government. However, as you just said, Nava works with all three levels of government, federal, state, and local. And given that Nava collaborates with all three of these levels of government, how does Nava as an organization select the projects that they build out for governments and their constituents?
[00:08:39] Sha Hwang: Yeah, that's a great question and we, we do think deeply about that. So we, in terms of how we select the projects that we bid on or, or work on, we think about it in the context of our theory of change.
[00:08:51] So in terms of what we're structured as a public benefit corporation to accomplish, we're trying to deliver for the government, simple and accessible service experiences that the public navigates and manages. We're looking to help the government deliver programs that effectively produce the outcomes for the populations they serve, and for Nava, we focus primarily on vulnerable populations, and we're also looking to support the production and the enablement of agencies that are adaptable in a world of continuous change.
[00:09:22] So that... those three facets broadly are our mission or our theory of change. And so when we look at work, we look at it first through that mission frame, and then we look at it through a kind of strategic perspective. And then we look at it through a business perspective. So if a new opportunity is coming in, we have a process that our business development team or our whole entire team uses that we talk about as opportunity assessments,
[00:09:46] We're saying, "okay is this project or effort, is it gonna help improve the service experiences? Is it gonna have a meaningful impact on a public's experience of a government program? Or is it gonna support that agency, adaptability or those types of aspects? And then if those answers are no, then we, we don't have to worry about, we don't have to worry about the dollar amount, the profitability, whatever, you know, all that sort of stuff. We can just screen, screen that work out.
[00:10:11] But then for the work that falls under that line, that is aligned with our mission; that is aligned with our theory of change; that is bringing us towards the world that we wanna see, then we think about the strategic perspective and there it has been pretty exciting to be able to work across federal, state, and local. Where, you know, we're now helping agencies that are working at a state level trying to also navigate the complexities of federal funding and compliance.
[00:10:36] Where there are sometimes conflicting direction or regulations around what states should be doing, even when they're working in good faith to deliver positive service experiences for their residents. But because we have some of the experience on both sides, uh, on the federal side and the state side, we can actually help them navigate that.
[00:10:53] And that from a strategic perspective is very interesting and valuable for us. When the lessons we can learn on one project or one effort, we can start to reapply or help agencies not have to learn those, learn those lessons the hard way in the future.
[00:11:08] Evan DeBroux: You know, you make a reference to making it easier...
[00:11:12] Sha Hwang: mm-hmm.
[00:11:12] Evan DeBroux: ... for people to access public benefits, there's a good segment of the population who don't think that topics like public benefits, transportation, they're not super, the "sexy" topics and also like, You think about public benefits and then you start thinking about its synonyms and it's like assistance and welfare and they, those can have very negative connotations and that that's just been historically true in the last couple decades. So how does Nava like attract talent given those negative preconceptions or how does not fight back on those negative preconception?
[00:11:49] Sha Hwang: Yeah, that's a great question. And it, and it's a sensitive topic, right? I do wanna be clear when we're talking about Nova as a public benefit corporation, we're a public benefit corporation, meaning that's our corporate designation. We have a social mission and social charter that's a part of our governance structure. It's part of our founding documents as an organization that allows us to make trade offs. You know, we don't have to just purely maximize shareholder value. We can make decisions that are in the benefit of our mission. Even if they are less profitable to Nava or that kind of thing.
[00:12:21] So that's, you know, public benefit corporation or like public benefit can be a loaded or multi-interpretation phrase there. But we do work with the government and we do work solely on programs that are benefiting on primarily these safety net programs, programs designed to serve vulnerable populations.
[00:12:38] Programs that oftentimes, as you're mentioning, can have stigmas associated with them, can be talked about as entitlements or things like that. You know, Nava as an organization, we are not the government, right? We are not setting policy; we are not dictating rules of eligibility or things like that, but the government is continually through legislation and communication is making promises to the public about what role it will play.
[00:13:08] It is saying, if you are in this financial situation, this is a program that is intended to help you through that period. If you are facing this health hardship and have this kind of household income, this is how we are trying to support you through that period. This is what you are eligible for.
[00:13:25] So the government is consistently making promises to the public through legislation, through regulations, through programs and policies and for Nava, we see our role as helping the government keep those promises that they've made; help them actually follow through on the things that they have committed to, to the public. For us, that's a very critical thing that the future that we want to see is that public institutions can continually earn the trust of the publics that they serve.
[00:13:54] And so we see our role as trying to support the implementation and the execution and the effectiveness of these programs. So in, in that context, I think that helps us be much clearer with what our goals are, right?
[00:14:08] And I think regardless of your feeling or capital "P" political alignments or perspectives on various programs, I think it's pretty under... under... I think it's pretty underestimated how much of our social and public life is supported by the infrastructure that the government is providing, but we do notice it really painfully when it's not there. Right? When the early phase of the pandemic in 2020, millions of people losing their jobs and overwhelming unemployment benefits, call centers and things like that, just calling hundreds of times, just trying to understand where their claims are.
[00:14:51] These are not creature comfort types of issues. This is not Facebook being down, so I have to check TikTok. This is, I need to pay rent. This is, I need to buy groceries. I need to keep my utilities on. You know, I... I need to survive.
[00:15:06] So I do think regardless of your alignment, you can understand some of the basic necessities that the government is promising that it will deliver to the public and some of the places where those promises have not been kept, or some of those gaps and the harms that are caused by that. So that's something that we think about a lot at Nava. We talk about a lot, and that I think has been able to allow us to talk pretty clearly about what brings people into this type of sector, what personal experiences they may have with these types of programs that allows them to understand it in a more concrete way.
[00:15:40] What abstract notions they might have they might not personally be experiencing some of these hardships, but they may understand the role and the necessity of them. I, I think that helps us cut through maybe some of the more divisive language around who deserves or who doesn't, or what's an entitlement versus what's, what's a benefit or that kind of thing.
[00:16:00] Ajay Jain: Awesome. So a Nava's homepage, y'all's website states that:
[00:16:03] "Nava's role in the growing civic technology space through quote, supporting structural change in government at every level is done in the following three ways.
[00:16:11] -Building services that are simple, effective, and accessible to all.
[00:16:14] -Helping programs meet the needs of beneficiaries and government staff.
[00:16:18] -And making agencies more resilient and adaptable in a changing world."
[00:16:22] So let's expand on that second point. Helping programs meet the needs of beneficiaries and government staff. Nav often combines its staff with government employees in order to solve a lot of the problems that government faces at a technological perspective. What is a thought process that your organization puts into building teams to tackle government problems both successfully and efficiently?
[00:16:43] Sha Hwang: Yeah, I appreciate that and that maybe to put it back, I think, you know, I like the civic technology Super Squad, right? That's a great way of putting it. I will say there's a lot of, I think, shininess or attention around civic tech or, you know, folks might talk about public interest technology or civic tech or gov tech or these types of things.
[00:17:03] But I think from our perspective, we see public servants as having been civic technologists for decades, right? A lot of the efforts, and this is something we talk about at Nava, one of the values we talk about is humility. And one of the values we talk about as a company is, is kind of building together, right?
[00:17:20] And the ways we talk about that are when we're entering a new space, like when we're... supporting or entering a team or starting to build a team together with the government, that we are oftentimes entering a conversation that has been going on for decades, right?
[00:17:35] Whether that's a legislative conversation, whether that's an organizational or institutional history type of conversation, there are folks on the public servant or civil servant side of things who have been keeping those fires burning for years and years and years before a team like Nava shows up.
[00:17:54] So many of the ideas or the impetuses or the pain points that are being observed or the solutions or paths forward that are being identified are not, I think, brought in by some kind of savior complex that Nava has, but are, I think, discovered and brought together and synthesized and empowered through the work that we do with our stakeholders and clients. And, and another way of thinking about it is like we think about our role as trying to be a conduit for the implementation of some of the ideas and energy and momentum that a lot of civic and public servants have been building for a long time before we've joined. In terms of team composition, I think we do think about it then from a pretty multidisciplinary lens, right?
[00:18:38] We're not just bringing in, we're not trying to bring in engineers that only care about the technical systems. We're not trying to bring in designers that only care about the polish of a specific mockup or that kind of thing. We're trying to bring in folks who deeply understand some of the user needs, and this, I, I would say this includes, for example, like case workers, field office workers, regional offices, you know, program directors, program operation, you know, all, all the... all the kind of staff that help make government function. We need to consider those needs and address those needs as well.
[00:19:12] But also thinking about not just from a pure technology only, or technology is the only answer kind of path to build these teams. One thing we've been doing more recently in how we've been building these teams that I think has allowed us to tackle some of these government problems successfully and efficiently is actually bringing in folks who have more of the programmatic expertise in the subject areas that we're working on.
[00:19:38] So for example, you know, we have worked over the last few years with the state of the Commonwealth of Massachusetts on their Paid Family and Medical Leave program. But in the early days, we were also bringing in folks who had experience with paid leave with early family programs like WIC, and understood very deeply some... some of the of the real life challenges of, of having a, of being a new parent, of needing to navigate these programs at that particular time, and being able to work hand in hand with the Massachusetts Department of Family and Medical Leave on the successful kind of implementation of that.
[00:20:13] So, you know, again, like I think maybe breaking it down into a couple aspects is like not just assuming, coming in, assuming that, you know, we're entering in on a blank slate, like, coming in assuming that there is a history that we're entering into and trying to deeply understand that history and context. And then from the similar side, trying to be able to bring that expertise and programmatic understanding ourselves as well so that we're not learning about some of the basics or just trying to think about our role as a siloed, like we're just... we're just the coders that show up or we're just the people making the mockups. We do take, take that partnership very serious.
[00:20:52] Evan DeBroux: So you make this mention about hiring SMEs and having the right tech talents and working on these projects and having people that have this sort of understanding of what your, what's the right word? Not, not vendor con uh, first, you're a contract. Part you're gonna contract from to work on.
[00:21:10] So, but I guess the next step is just sort of implementation of these tech projects or just these projects, period. I mean, it's, technology isn't always answer you... you referenced that in your previous answer. So how does Nava utilize these agencies, adaptability and resiliency, as well as other methodologies in order to achieve the outcomes Nava aims for?
[00:21:36] Sha Hwang: You know, some of that aspect of our theory of change goes back even to some of our experience on healthcare.gov, right? When we were some of the lessons that we built from healthcare.gov in terms of, we did work on a lot of the redesign of the application and, but you know, no matter for example, how good or how user centered or how streamlined the application would be,. If it took a really long time to get a security audit approved; if it took a really long time to get a cloud production instance stood up; if it took a really long time to get agency staff trained on some of these new aspects, these things, these kind of agency adaptability aspects could really stall out even the best intentioned modernization efforts or efforts to improve the kind of customer or user experience.
[00:22:24] So, one of the things that we've been working on for the past few years is an effort with, um, the Center for Medicare and Medicaid Services, who's the agency that runs healthcare.gov on some of their internal cloud platform, and that's called like CMS cloud or cloud.cms.gov. And some of the goals there were, you know, how do we bring in, you know, Nava you know, our mission?
[00:22:46] We don't have to do bid on all the contracts. We don't have to win all the work, but how can we work with CMS and support them to make sure that any new team that they're bringing on, whether or not it's a internal government team or an external contractor team, can get a new production environment within 48 hours.
[00:23:06] How do they go rather than six months of approval, 18 months of approval waiting for these provisioning, you know, how do we get them so that if they need, if they're identifying a need, that the team can be onboarded within 48 hours. And, and those are, that's been amazing for us to be able to see and being able to support.
[00:23:25] Now, CMS, second, no downtime, open enrollment across healthcare.gov and medicare.gov. These are I think, pretty exciting ways that we can see some of the outcomes of this improved resilience or this improved kind of developer experience.
[00:23:40] And I think we see examples of both successes and failures in other agencies, right? Like I don't. know all of the specifics of, for example, some, the rollout of the recent executive... Biden's executive order on, on student loan cancellation. Uh, but you saw right in that period you saw studentaid.gov go down very quickly due to the amount of load and the amount of interest. And so these are things that where we see some of the, when we talk about resilience, what we're talking about is trying to, what are the systems that can be put in place? Not just user experience or that kind of thing, but what are the infrastructure or operations or process things that can avoid those types of outages or, or kind of fragility in the future?
[00:24:25] And I think we also see positive, I, I don't want to single out studentaid.gov. I think we also see positive examples, right, where you see like covid tests do gov or the partnership with USPS, where that roll out, you know, a very broad audience as well, A lot of interest, but in terms of the way that they architected it or the way they set it up, they were able to handle a very, very high amount of load with that level of planning and thinking about that kind of resilience.
[00:24:50] Ajay Jain: Yeah, so, I completely agree with that as well. I remember when that whole COVID test for all thing happened at the USPS, I was also like going through my Twitter feed cuz my Twitter feed basically just consists of travel content and people who I follow in the civic tech space and like political correspondents. And I think I was surprised to see how seamless of a overall out it was in terms of like a civic technology perspective. Just because like, those things don't happen that often, and especially with something as big as like requesting free COVID tests at a point when COVID was going through a be a brief spike last year was really, really cool to see.
[00:25:23] And I think it was a really prime example of the ways that building a team together in the civic technology space can impact hundreds of millions of Americans in a positive way.
[00:25:30] Sha Hwang: I think that's, and I, I want to touch on like two aspects of that, and again, this is. Shout out to all the teams that worked on it, and I, I know, I don't know of the folks personally, but I know for a fact that it probably required a ton of work and a ton of thinking. But on healthcare.gov, we, we identified one of the main failures sources of outages was the login system. So we built a replacement login system that we called the Scalable Login System because we're incredibly boring with names.
[00:26:00] But... but we, before rolling it out, load tested it with a billion users and the goal of that load test was just to quell any doubt or any concern that this new system developed by a very small team like, like six, six engineers, could actually withstand some of the load that was plaguing a whole bunch of other systems.
[00:26:21] So like that was, you know us in our corner, thinking ahead, doing some of the load testing, trying to pull that through. So I think like obviously the covidtest.gov team thought about that and did some load testing or did some evaluation of their architecture or of their cloud infrastructure that allowed them to feel, well, hopefully somewhat confident that they could handle some of the high load.
[00:26:47] But I also wanna point out, I think a service design aspect, which I, I really appreciated, which was, you know, where I, when I say like, technology isn't always the solution. Right? You know, one of the decisions that was made that I'm sure was not easy to get approved was that no one had to create a USPS account. No one had to create, go through some sort of verify that I live at this address, you know? Do create a username and password, all this stuff, all of which would've reduced the number of people who would've gone through that funnel. All of which I'm sure could be justified for various reasons, political or program integrity or otherwise.
[00:27:24] But, In reducing those steps and actually just saying like, this is, this is just a form. You're just gonna check out a thing that has a $0 amount, you're just entering in your address. We're gonna do a very basic level of kind of verification. I think that that was a... that was not a technical decision, right? It was a kind of service design or business process decision, but it had huge impacts, I think, on the success of a program or effort like that.
[00:27:51] Ajay Jain: Right, and I think it kind of actually goes back to kind the lessons that civic technologists learned from designing healthcare.gov and like all the, you know, the ramifications of those things that happened when healthcare.gov went down, when it first went live several years ago. And taking those lessons and like adapting 'em to like future civic technology projects. And the ones on top of that and the ones on top of that, similar to what you kind of described earlier, at the start of our recording. And kind of in a way almost culminating in like these like seamless civic technology experiences where we still have lessons to learn like with Biden's announcement of student loan forgiveness.
[00:28:20] But we also have, you know, areas right now where, for example, with COVID tests where it is a lot better and a lot more impactful than it was before, which I think is, is really cool. Actually, I briefly wanna segue away from this to a point that you had made about load testing. And I'm actually really curious from like a technological perspective when technologists such as, such as y'all are testing like healthcare.gov.
[00:28:40] And I guess in this situation, although you wouldn't know it since you guys didn't work directly on the PS rollout, how do you guys do load testing in order to ensure that when you have hundreds of millions of Americans who are accessing these websites that they don't go down?
[00:28:54] Sha Hwang: Yeah, I think there's, there's varieties of, I think, tools and practices in order to generate load tests. And, you know, generally what you want to do is, you know, load tests in, in general are, are different than, for example, integration tests or unit tests. And you would say like, maybe a unit test is, you know, does the, does the doorknob turn? Sorry, I'm looking at my, a door right now. So that's, that's where I'm going you know.
[00:29:19] A unit test is, "does the doorknob turn when you put your hand on it? Does the door open when you pull the doorknob?" An integration test is, you know, "can I reach that doorknob, turn the handle, and open the door and walk through?" Great, You know, you can complete the process of entering the room, but load testing is, you know, "what happens if I simulate a hundred thousand people walking through this door? Does the doorknob fail first or does the hinge, or does the, Is the door wide enough to fit all the people at the rate, or is the room that I'm sending people to full?"
[00:29:57] You know that, that that's when you start to answer those questions. And I think it's just a healthy thing to do in any new production, new production system or environment, how we modeled it for healthcare.gov was taking the maximum load that we had seen and then modeling 50 times that. So getting to a level of traffic, then loading up our system with a billion users and seeing. You know, at the time I think healthcare.gov had about 12 million users. So even at multiple orders of magnitude difference, could it still withstand 50 times the traffic, the live traffic of what we had ever seen at peak on healthcare.gov.
[00:30:34] So like trying to ensure that we are structuring the systems and learning what is breaking. Is it the database that's filling up? Is it the API endpoints that are taking too long to respond? Is it the application server that's gonna fail in some transaction? Like really being able to understand that we can learn a lot from that practice.
[00:30:52] And then I think also thinking about how to structure the outcomes, being able to actually look at what type of test we were doing. Being able to load up a system in that way, and then being able to test it against a simulated amount of traffic that was far above the peak that we saw on healthcare.gov and doing that for a sustained amount of time.
[00:31:14] I've seen other firms, and I won't name names, but that, you know, think doing a load test for a minute is gonna be enough. You know that that's not, you know, being able to conduct that load test for an extended period of time. You really wanna see whether stuff will fall down. Right? You're not doing it to, to kind of hold some pride. You're doing it to learn because you wanna learn when it's safe rather than when it's real people trying to access a real service.
[00:31:39] So those are the ways that we, I think, think about load testing. And then there's other ways I think, you know, again, from maybe a more service design perspective or business process perspective, there's ways to de-risk things on the other side, right? Rather than for a, a recent rollout of a new public program, what we did instead of, you know, when the, when this, when this program is legislative, really required to be available for residents of this entire state, that they all can access it all at the same time, and then we do some big bang press release and all this stuff.
[00:32:13] Rather than doing that, what we did, um, in one of these cases was prior to the legislative deadline release, it only to people who had a certain level of credentials and access. So, rather than an integration test, like one person going through the room and testing that we could at least have kind of like a, a soft launch or a, you know, soft opening type of thing. We have more people over, have people go through the system a few times, have people with different circumstances and different cases before we then, you know, roll out to the public and, and ga So we've been able to kind of address some easy kind of user feedback or things like that.
[00:32:52] So that that's another way of, you know, that's, you know, a different layer of abstraction than like load testing the system itself, but is like thinking about how we're doing, how we're avoiding a kind of big bank launch at all, and reducing the stakes for that day zero or day one experience.
[00:33:09] Ajay Jain: I honestly love that description that you had just now of like stress tests or like of, of these tests, these load testing, right. Because it reminded me a lot of like stress tests, like back in science, back when I used to be in high school and we do Science Olympiad you had like bridge building competitions, "like how much weight could the bridge like undertake before it like collapse?" And you obviously want the bridge to meet a certain amount of weight, so you would beat other teams, but in this case it's a lot different because your bridge is, you know, a project that millions of Americans are going to be using. And you wanna make sure that it holds the weight of all of these people trying to use the website all at once when it first launches.
[00:33:40] And part of that problem is also kind of segued away by doing like a soft launch and like just kind of like a beta testing like period before getting to that, that full hard launch, which I think is also really, really cool. And definitely something we didn't really have in Science Olympiad competitions when it came down to the get go.
[00:33:54] But actually this. To another question of mine. Back when I used to work in political tech, I would often see this infographic on Twitter, and I still see this infographic even though I'm no longer in political tech about different political technology organizations and their roles in progressive politics.
[00:34:09] So for example, Civitech is used for voter engagement and volunteering analytics while a company like Act Blue is used for processing donations. And similarly, and we've kind of touched on this earlier on in our recording, there are a ton of individuals and organizations in a civic technology space with different roles as well in terms of tackling government solutions. What is Nava's role in this space? Who are the partners that the organization impacts, and what does your organization produce?
[00:34:35] Sha Hwang: I like to take a very broad way of defining civic tech. I think there are some folks who may talk about civic tech as a community or as an ecosystem, purely as the organizations building kind of outside participation or outside visibility into inside like government processes or programs or things like that.
[00:34:56] There are other folks that might talk about public interest technology or gov tech or things like that. I don't know. I think for us at Nava, we see a very broad tent of all these various stakeholders from folks doing policy and program research; folks doing impact analysis and program evaluation; folks doing the call center operations; the kind of case workers, the paper processing that goes into some of these programs, the field office workers.
[00:35:27] All of these various organizations all up and down, kind of the layers up, up and down the stack. In terms of layers of abstraction for us, as I mentioned before, I think we see our role as the people who help, uh, the, you know, the people who produce the successful implementation of policies or programs that are going to serve vulnerable populations.
[00:35:52] And so we are not, for example, making, we are not trying to play a TurboTax-type role, right? Where we are creating an alternative path to accessing government services or government programs or an a paid third party way of accessing government programs.
[00:36:11] We're trying to work on and work with government agencies on delivering an excellent first party experience. Delivering that actually streamlined user experience to access that benefit rather than a kind of outside path or that kind of thing. So there are other ways to talk about it. I, I will say, like you mentioned like in political technology, Nava as a government contractor. So we do work directly with government agencies.
[00:36:33] We're not a partisan organization. We're not working with political parties or that kind of thing. We are, we're not, as I mentioned before, we are not the makers of promises, we are not trying to play the government role. We are trying to support, support and empower the government agencies in order to have the successful kind of execution and implementation of those efforts.
[00:36:55] Evan DeBroux: Nearing the end of our questions here. So we wanted to take the opportunity to discuss the well recent task slash present and the future for Nava. So can you discuss with us at least, least what stuff that's been released? How, I mean, you've talked about healthcare.gov and you've talked about WIC*which I'm, I believe is a more recent project. But, how has Nava used technology to increase access for public benefits? And what sort of obstacles did you run into on sort of your recent projects to overcome, basically overcome issues with access to public benefits?
[00:37:35] Sha Hwang: Yeah, that's a great question. We did, as I mentioned, we have worked with the Commonwealth of Massachusetts for the past few years on helping them implement and roll out their Paid Family and Medical Leave program, which is frankly, an amazing benefit. And actually, you know, Nava's also an employer in Massachusetts, so we got, you know, the notices to be become kind of enrolled in that program and things like that. So it's been exciting to be both a user from a organization standpoint and to be a team that's working to support Massachusetts DFML, the Department of Family and Medical Leaves on that.
[00:38:10] I think there's various things that we brought, you know, in terms of lessons that we brought from other eligibility and enrollment experiences from, for example, healthcare.gov or doing state level work in Vermont, we had integrated all of the health and economic programs in Vermont to use a single shared module. So like using a single shared kind of code base around how to upload and verify documents. All of these various health and economic programs could use have that similar step in their processes, but rather than buying and building that 50 different times, I think it was 37 different programs actually, sorry.
[00:38:47] Rather than buying and building that independently 37 different times, you could just build it once and reuse it. So, bringing some of those lessons into supporting and delivering the Commonwealth of Massachusetts's Paid Family and Medical Leave program.
[00:39:00] But something I think that's been exciting in the, in the rollout and delivery of it is, one, the number of families that have been able to access this benefit. I think it's in the first year it was something like 70,000 claimants were able to be processed with something like about $700 million^ in benefits being dispersed to those claimants. So the scale is pretty amazing in terms of being able to support, support these populations in Massachusetts.
[00:39:29] But what I think has been incredible and all really all the credit goes to DFML for this is, you know, based on some of our user research, we also saw that I, I don't know if either of you are parents, but I, I am, I'm a parent of a four year old, four year old daughter, and most of these paid family and medical leave programs it's exciting to see states start to roll them out.
[00:39:51] But, pretty much all of them, you are only eligible once, you know the, once the child arrives. So you are not, you cannot apply for this program before the birth of a child. So, and then you have maybe a window to apply for that... navigate this government process to be able to qualify for the benefit to be able to receive some of the, uh, compensation or kind of percentage of income or that kind of thing.
[00:40:18] In our user research, as you might imagine, that is not the, the easiest time to think about navigating a government benefit program. That is, there are other things on people's minds that they're prioritizing and it's very difficult for employers to contact. There are employees who are out on leave or get the right paperwork involved and get the... get the right forms in place. So DFML I think actually updated some of the regulations based on the research that we learned, that you can actually, as a new parent, you can apply for this bene prior to the birth of your child and the employer because you know, the HR is in contact with this employee the whole time can, can kind of verify this claim, you know, independently.
[00:41:02] And that's just like a huge burden reduction for new parents rather than, you know, in the hospital needing to think about what forms you need to fill out for your employer, that kind of thing, but actually being able to do that in advance of your leave. So I think that's like, it's a personal one for me because I can very deeply empathize with that situation. But I think it's, it's really just a testament to DFML being very kind of user-centered and being able to listen to the participants and claimants in the program. Being able to understand employer needs and being able to actually work to update the business processes to allow for this benefit.
[00:41:39] Ajay Jain: Can you share your perspectives on where you see the expansion of opportunities for Nava in the digital service phase as governments require more technological resources?
[00:41:47] Sha Hwang: That's a great question, and I think. I think less about, you know, the opportunities for Nava specifically, because I think for us, again, like we are here to support the agencies that are running the programs that affect the populations that we're here to serve. That's, that's what we're here to do.
[00:42:07] But as I mentioned before, there are also different actors in the ecosystem or different, different ways of using technology in the relationship to public services or public programs that are pretty complex, right? That are pretty, pretty nuanced and challenging. So there are, I think from our perspective, there are some services which I, I believe in, I think Nava believes are in the public's best interest to remain public. Meaning, when I talk about this is why we explicitly are not trying to build our own product or licensed platform, or a third party path to access these types of services because we, we deeply believe that some of those, uh, while that can be done; while that can be possible; while that might even be a very attractive kind of investment for venture capital or things like that, that the logical end points of that, or that the incentives that that creates are divergent with the need of the public.
[00:43:10] And so one very recent example is identity verification. So, I don't know if you've followed, but I think at the beginning of this year, irs.gov announced that they were going to require all new accounts to be created with a service called id.me. id.me is a private company. They are handling identity verification for a variety of private and public services. They, they provide a licensed product to provide these identity verification services and irs.gov was telling all new users saying, "you need to register to create an account on irs.gov, you need to do that through, id.me and id.me will kind of broker the tokens to be able to verify that access."
[00:43:55] What id.me was also doing in that was also then as a part of their intentions to streamline identity verification, they were also requiring facial recognition or facial scanning in order to confirm your identity. So, you would upload photos of your driver's license or passport and then also open up the camera on your phone or computer and they would take a scan of your face and kind of verified the two.
[00:44:22] And I think that's, there's a whole set of, we can talk about this for a long time, there's a whole set of nuances and conversations to go into there, but I, I do think there are access issues with a private, private service owning some of this public information. There are equity issues with who then is not able to access some of these basic programs because of some burdens being placed through other parties and other vendors.
[00:44:51] So I do think in terms of the future or these expansion of opportunities, I think there's a whole host of programs that could be built better. I think there's a whole host of eligibility requirements that could be made more human-centered as DFML has done in terms of their program. I think there's a whole, like I, I think there's a lot of energy, for example, and excitement around Biden's recent executive order around reducing administrative burdens and enhancing the customer experience. These are all very exciting to me in terms of new territory or more clear ways to measure successful outcomes in public services and public benefits. But when they intersect with technology, I think there's also real nuances around who is building, who owns some of this data, who is building it? Where do those incentives fly? Where does the ownership lie? Where does accountability lie?
[00:45:43] And there's, in all of those questions, Nava has kind of made our choices to structure ourselves in a way where we don't have to ask some of those questions, but I do think it's something we as a community or as a broader set of organizations, need to be pretty critical of because they have massive, massive implications years down the line.
[00:46:02] Right? We are living in a state now where, the credit reporting organizations like TransUnion, Equifax, and Experie Experian are those identity brokers or data brokers for their private entities that are used for all sorts of public verifications or things like that. So, we need to be very cognizant of, I'm less saying that's good or bad, but I'm more saying that there are long-term implement... implications about this intersection of the role of the public or the role of technology, and where folks in this space sit, and I do think it's something that we need to take, take on pretty, pretty carefully and pretty considerately.
[00:46:38] Ajay Jain: Sha, thank you for coming on today. This is a really, really awesome conversation that we got to have about Nava in its role in the civic technology space and some of the problems that y'all are trying to solve. This is really awesome.
[00:47:08] Thank you again for taking the time to tune into this episode. We hope you found this conversation with Shaw as intriguing as we did. Do share this with any friends, family, coworkers or neighbors interested in learning about Nava PBC's work, and a tech side of public benefits.
[00:47:23] Evan DeBroux: As always, please make sure you're following us on our Twitter handle @thinkcivic to join in on the conversation after this episode. Be sure to hit subscribe at thinkcivic.substack.com. Thanks everyone!
[00:47:38] Ajay Jain: Bye, y'all.
Footnotes:
*Evan meant to mention their work with the Department of Paid Family and Medical Leave (DFML) program, not the Women, Infants, & Children Nutrition (WIC) program
^Sha’s around $700 million in benefits number is actually a slight underestimate of the actual figure of $777,497,058.43
Sha Hwang: Nava PBC