Signals of Readiness: How Funders Can Vet AI in the Social Sector

The SMS service that connects hungry neighbors with nearby food pantries? It runs on AI. The nonprofit fighting cancer by repurposing generic drugs? Also operating with AI. The nonprofit instantly translating between teachers and parents who speak different languages? AI again.

Contrary to the dominant headlines, AI isn’t just a tool for big tech. It’s already fueling social impact, bringing benefits to under-resourced communities around the globe. Nonprofits, often overlooked in the tech world, are putting AI to work in deeply human ways.

A recent study by Fast Forward and a team of academic researchers revealed that 60 percent of AI-powered nonprofit respondents have been using AI at the core of their solution for more than a year. But they can’t do it alone.

Research conducted with the Annenberg Foundation found that their grantees want the following forms of support for their AI adoption:

• Strategic advice on values-aligned AI and data practices
• Training to equip teams with the skills and confidence to design and assess AI tools
• Grants for tools to seed hands‑on experimentation in real‑world contexts

Nonprofits have the ideas and the drive, but they often lack the capital to execute. That’s where grantmakers come in. But grantmakers need better ways to evaluate AI. We need frameworks to understand how nonprofits are thinking about the data behind the models, the design decisions driving outputs, and the community voices shaping the tools. AI’s outputs are only as good as the data and practices behind them. As grantmakers, we have a chance to identify and invest in what’s working.

From our perspective as grantmakers focused on unlocking AI in the social sector, we recommend starting by asking strategic questions of potential grantees. By doing so, you will surface signals of readiness to scale, risk of harm (and plans for mitigation), and opportunities where targeted support could unlock greater impact.

In the sections below, you’ll find a checklist of questions that grantmakers can ask to vet whether a nonprofit is thinking critically about building responsible AI. These aren’t meant to gatekeep innovation. They’re meant to reveal the organizations doing the hard work, and send a clear signal to the broader sector: if you’re deploying AI in vulnerable communities, this is the work.

Curating Better Datasets

Strong AI systems start with strong datasets. Many nonprofits have accumulated datasets, but they are often incomplete and stored on legacy systems. Luckily, nonprofits are innovative. They are finding creative ways to improve the quality and structure of the data they use.

Nonprofits can drive impact by leveraging secure, highly relevant data — often from their own users — that reflects lived experience. By using this data to supplement AI, they can generate outputs that are more practical and context-aware than those generated solely by the foundational models that many of us use every day.

Tarjimly models this practice. Tarjimly is an AI-powered nonprofit that connects refugees and immigrants with real-time translation support. To improve the accuracy of their AI-driven language tools, they built a custom training dataset sourced directly from their community of volunteer interpreters.

Many of these interpreters come from the same communities as the users, and they shared real examples of how people actually speak. That input was then structured and cleaned to train models that better reflect the lived experience of the people Tarjimly serves. It has a dual effect: It not only provides a large corpus of information to generate accurate AI outputs, but it also emphasizes linguistic nuances specific to refugee communities — data that would never show up in mainstream commercial models. It’s a form of “community-powered fine-tuning” that both reflects and serves their unique user base.

Other nonprofits are focused on cleanup. They’re removing outdated records, standardizing formats, and filling in missing fields. This work is complex, costly, and time-intensive, which is why philanthropic support is critical. For example, Scrutinize uses AI and data analysis to process New York state court records. They clean and structure judicial data to make judge profiles accessible for public oversight. By turning opaque legal decisions into clear, searchable insights, Scrutinize helps voters and advocates hold judges accountable.

Questions grantmakers can ask about representative datasets:

• Where does your training data come from?
• Are the communities you serve represented in the data you’re using or generating?
• What steps have you taken to clean, structure, or validate your data?

Testing for Bias in AI

Before we direct resources, we have to know what success — and harm — actually looks like in an AI context. This is where the conversation about bias begins. But bias isn’t exclusive to machines. Human decision-making is deeply biased. So, the question isn’t just “is the AI biased?” It’s “how does this AI system compare to the bias in our current state?”

To answer that, we need to examine how these tools perform in practice — and that’s exactly what nonprofits on the frontlines are doing.

One way to do that is scenario testing: targeted prompts designed to uncover and correct for bias. For example, does the tool default to a college pathway, even when that’s not the best option? Does it assume users are straight? Does it have a point of view on an ideal family structure? These questions surface subtle harms before they scale. And this can’t be a one-time check. Bias testing should be a regular part of any AI-powered nonprofit’s workflow.

Another practice is an outcome analysis. This means auditing which recommendations are given to which users and identifying patterns that may signal inequity. Are users from certain zip codes consistently routed toward lower-resourced opportunities? Are women steered towards roles that pay less? This kind of analysis helps nonprofits spot systemic patterns and intervene early. Systems behave differently depending on who built them, what data they were trained on, and how they’re tested. That’s why regular testing and evaluation is essential.

Quill, an AI-powered literacy nonprofit, builds ethical considerations into every step of their process. Their training data comes from real student work and teacher feedback, curated by educators to reflect classroom realities while ensuring privacy is never compromised. Before launching any new activity, Quill runs three rounds of bias testing, including scenario testing with their Teacher Advisory Council. They also review more than 100K student submissions a year to detect patterns — like certain student groups getting lower-quality feedback.

When their AI performs inconsistently across student groups, they adjust the dataset and re-train. Their goal is to achieve coaching that mirrors a veteran teacher, delivered fairly across every classroom.

As grantmakers, our role is to surface not just where AI might reinforce bias, but how it compares to the human systems it’s replacing. That means asking sharp questions to uncover patterns, test assumptions, and understand how equity is — or isn’t — being built into the AI system from day one.

Questions grantmakers can ask about testing for equity and bias:

• Do you regularly test your AI outputs for bias or unintended outcomes? How?
• What specific types of bias are you watching for?
• What systems are in place to flag and fix bias as it’s discovered?
• How does your AI system compare to traditional decision-making approaches in terms of consistency or fairness?
• Has the AI helped reduce inconsistencies or bias present in your previous human-led processes?

Soliciting Community Feedback

CareerVillage models both approaches. When they built Coach, their AI-powered career guidance tool, they didn’t go it alone. During the beta phase, they partnered with more than 20 youth-serving organizations, including Year Up, AVID, and Big Brothers Big Sisters chapters. Together, these partners submitted more than 700 pieces of feedback. That input shaped everything from tone and language to the structure of Coach’s activities.

But feedback didn’t stop at launch. CareerVillage also has an in-product feedback mechanism, allowing users to give real-time feedback on every AI-generated response, and the team makes changes quickly and publicly. The result is a product that gets better and more inclusive with use.

Wharton’s Human-AI Research team found that real-time user feedback, especially when paired with transparency about how the system learns, increases trust in AI tools, particularly chatbots. The same is true in the social sector: feedback builds legitimacy.

Community feedback isn’t a “nice to have.” It’s essential infrastructure for AI that works for beneficiaries, nonprofits, and grantmakers.

Questions grantmakers can ask about community feedback:

• Were beneficiaries or ecosystem partners involved in designing or testing the tool?
• Can users provide real-time feedback on AI-generated responses?
• What’s the process for incorporating feedback into product improvements?

Grantmakers don’t have to become AI experts, but they do need to ask the kinds of questions that will surface red flags and positive signals — and they need to know how to read and understand these signals.

Shannon Farley is co-founder and executive director of Fast Forward. Chantal Forster is an independent advisor with Warren West Advisory and former executive director of the Technology Association of Grantmakers.