Evaluating Legal AI Providers Across Trust, Privacy, and Security
The questions legal teams should ask to understand how AI providers protect data, disclose tradeoffs, and maintain trust as their products evolve.
Law firms and in-house legal teams evaluating legal AI providers face a different set of questions than most enterprise software buyers. Lawyers must ensure the confidentiality and protection of their client’s data at the same time while also keeping up with the pace of change. Therefore, one of the most crucial questions that buyers and innovation officers must ask is: can their legal AI provider innovate at the speed that AI demands whilst still prioritizing and keeping their customers’ trust, security, and privacy?
An example from a couple months ago illustrates why this matters. When Anthropic released Claude Fable 5, it also announced that the model required additional safety measures. Unlike Anthropic's zero data retention (“ZDR”) offering for many of its other models, Claude Fable 5 requires 30-day data retention and permits limited review by Anthropic's safety team for interactions flagged by its safety systems.
That left us with a decision. Should we delay adopting one of the most capable models available because it didn't support ZDR? Or should we make it available while being completely transparent about the change in data handling?
We chose the latter. Harvey made Claude Fable 5 available as an optional model, clearly and very publicly, disclosing that it goes against our standard practice of offering models with ZDR. We allowed customers to decide whether the additional capabilities justified that tradeoff for their use case.
What we didn't do was incorporate a model with weaker data protections without telling anyone. That should never be an option.
This scenario got me thinking about how other providers handle moments where there is a tension between innovation, speed, and data protection, and whether they're as forthcoming. Not all of them are, especially when it comes to:
- The potential for government access to customer data;
- Data localization commitments and whether true localization can actually be achieved;
- How legal data is sourced;
- Whether ZDR commitments continue to apply as AI products evolve beyond simple chat interfaces into long-horizon agents and long-running workflows.
The rest of this post is a practical guide for legal teams, CIOs, procurement leaders, and security teams evaluating legal AI providers, offering specific questions to ask and red flags to watch for.
A Practical Checklist for Evaluating Legal AI Providers
The checklist below walks through key questions to ask across data sourcing and IP, data localization, ZDR, subprocessors and supply chain, and governance and accountability.
Why Transparency Matters More as Legal AI Evolves
When we talk about customer trust, we're talking about more than just the privacy and security of our customers' data. We're building a company around the idea that our customers trust us to be honest, transparent, and to not brush difficult topics under the rug. That's the real differentiator: Does your AI provider proactively communicate with you and give you the information you need to make informed decisions? Are they willing to address the difficult topics other providers won't talk about?
In a recent subprocessor update, we proactively addressed the issue of prompt caching and how it works with zero data retention (ZDR), which is something a lot of providers stay quiet about. That update is available for our customers to read here.
Looking ahead, buyers should evaluate not just a provider's current commitments, but how they communicate and adapt as AI continues to evolve. The question worth asking: Is transparency their north star?
Trust is a Crucial Part of the Platform
For legal organizations, evaluating an AI platform is not simply a question of product quality or model performance.
The real test is how a provider makes decisions when trust, privacy, governance, and compliance create constraints on what is possible. Rather than avoiding those constraints, the strongest solutions make deliberate choices, explain them clearly, and remain accountable as technology evolves.
These issues are complex, evolving, and rarely black and white. We do not claim to have solved every challenge. But we do believe legal AI providers should be willing to confront difficult questions directly and be transparent and accountable in their response. The easier path would be to ignore them, but that path only leads to a damaging loss of trust for the industry as a whole.
