The Real State of Accessibility Testing in 2025
Artificial intelligence promises to make everything faster, smarter, and more efficient. For developers and business owners working on web accessibility, that promise is tempting. Imagine an AI that could instantly scan your website and flag every single accessibility issue, giving you a clear path to WCAG and ADA compliance. It sounds great, doesn’t it?
Well, we’re not there yet. Not even close.
Recent tests in October 2025 show that even the most advanced AI tools only catch about 57% of accessibility problems. That number, while a huge leap from older automated tools, hides a more complicated truth. The AI is good at finding the simple stuff, the black-and-white code violations. But it completely misses the issues that cause the most frustration for people with disabilities, the ones that require human understanding, context, and experience to find.
So, where does that leave us? We decided to put three of the biggest names in AI to the test: OpenAI’s ChatGPT-4o, Anthropic’s Claude 3.5 Sonnet, and Google’s Gemini Pro. We didn’t just want to see what they could do; we wanted to find out what they couldn’t. The results might surprise you.
The Great AI Accessibility Test: Our Methodology
To get a real sense of how these AI tools perform, we couldn’t just throw a few lines of code at them. We needed a realistic, challenging, and consistent way to measure their performance against the gold standard: a human accessibility expert. Here’s how we set up our experiment.
100 Real Websites, 300+ WCAG Violations
Our test group wasn’t a curated list of perfectly coded examples. We picked 100 real-world websites. This included e-commerce stores with complex checkout flows, small business websites with tight budgets , and content-heavy blogs. Before the AIs ever saw them, our human expert performed a full accessibility audit on each one, identifying over 300 distinct WCAG violations. This gave us a verified list of known issues, from simple mistakes to deep-seated user experience flaws.
Human Expert vs. AI: The Baseline Comparison
The entire test hinges on one principle: a human expert sets the baseline. An automated tool can give you a score, but only a person can tell you if a website is truly usable. Our expert used screen readers, navigated with only a keyboard, and assessed the logical flow of information, all things that AI currently can’t do. The expert’s findings served as the “ground truth” for our comparison. The goal wasn’t just to see if the AI found an issue, but if it found the right issue for the right reason.
What We’re Measuring (And What We’re Not)
We gave the AI models code snippets, screenshots, and descriptions of user flows from our test sites. Then evaluated their feedback in several key areas: code correctness, alt text generation, form accessibility, and content structure. We weren’t interested in a simple “pass” or “fail” from the AI. We wanted to judge the quality of its analysis. Did it understand the context? Did it explain the human impact of the problem? Or did it just repeat a generic rule from the WCAG documentation? A 100% score from a tool means nothing if a user with a disability still can’t use the site.
The Hard Truth: AI Catches Just Over Half the Problems
The headline statistic from our testing is that AI-powered tools automatically find about 57% of total accessibility issues. On the surface, that sounds pretty good. It’s a marked improvement over older, non-AI scanners and provides a fast first pass for any team. But this number needs a closer look.
The 57% Figure: What AI Can and Can’t Do
The 57% refers to the portion of WCAG rules that are considered “machine-testable.” These are binary, rule-based checks. An accessibility checker can easily tell if an image is missing its alt text, if a page is missing a title, or if you’ve skipped a heading level (e.g., jumping from an H2 to an H4). It’s checking for the presence of an element or a specific attribute in the code.
For development teams trying to establish a baseline for WCAG compliance, this is a great start. Running an automated scan can catch dozens of low-hanging fruit in minutes, saving a ton of manual review time. But it’s a mistake to stop there.
Why the Other 43% Matters More
The 43% of issues that AI misses are almost always the most serious ones. These aren’t simple code mistakes; they are failures of logic, context, and user experience. An AI can confirm that an image has alt text, but it can’t tell you if the text is a helpful description or a meaningless string of keywords. It can verify that a button is coded correctly, but it can’t tell you if the button’s purpose is confusing.
These are the barriers that truly prevent someone from completing a task. Think of it this way: an AI can act like a proofreader, checking for spelling and grammar mistakes. But it can’t tell you if your story is boring, confusing, or doesn’t make sense. The issues AI misses are the ones that determine whether your website is just technically compliant or actually usable for a human being.
ChatGPT-4o Performance Analysis
ChatGPT is often the first AI people turn to for quick questions, and its performance in our tests reflected that. It’s a solid starting point for general queries, but it quickly shows its limitations when faced with the nuances of web accessibility.
Strengths: Good for Basic Code Checks and Explanations
If you need a fast explanation of a WCAG rule or a basic ARIA attribute, ChatGPT is a helpful conversational partner. You can paste in a snippet of code and ask, “Is this accessible?” and it will generally give you a clear, easy-to-understand answer. For developers who are new to accessibility, it acts as a decent educational tool, breaking down technical concepts into simpler terms. It’s useful for catching obvious errors in isolated components before they get integrated into a larger system.
Critical Misses: Context and User Experience
The biggest problem with ChatGPT is its lack of context. It often provides advice that is technically correct but practically useless. For instance, when asked to generate alt text, it sometimes produced descriptions that were just a list of keywords, failing to capture the purpose of the image on the page.
We also found instances where it provided outdated code examples or suggested fixes that only partially solved the problem. It can’t understand the user’s journey. It won’t know if a series of technically correct steps leads to a confusing or frustrating dead-end for a screen reader user. Relying on it for a full accessibility audit would leave massive gaps.
Claude 3.5 Sonnet Results
Anthropic’s Claude 3.5 Sonnet came across as the more thoughtful and systematic analyst in our tests. It often went beyond the technical rules to explain the human side of accessibility, though it wasn’t without its own set of limitations.
Best for: Complex Form Accessibility
Where Claude really stood out was in analyzing complex user flows, especially in forms. When we presented it with the code for a multi-step registration or checkout process, it was better at identifying logical gaps. For example, it could point out that an error message wasn’t clearly associated with its form field, or that the tab order would be confusing for a keyboard-only user.
It excelled at explaining the “why” behind a recommendation, framing its advice around inclusion and user experience. This makes it a valuable tool for educating team members who might not be as familiar with the real-world impact of accessibility barriers.
Limitations: Color Contrast Edge Cases Despite its strengths in logic, Claude had trouble with some visual nuances. It can’t “see” a website, so it has to rely on the code. This means it can miss issues that are obvious to a human inspector. For example, it struggled with color contrast problems, especially in edge cases involving gradients or text over images. While it could calculate the contrast ratio between two solid hex codes, it couldn’t reliably assess the legibility of a button whose background was a busy photograph. Its alt text suggestions were also frequently too long and descriptive, requiring a human editor to make them more concise and useful.
Google Gemini Pro Findings
Google’s Gemini Pro brought a unique strength to the table: its powerful multimodal capabilities. Being able to analyze images and screenshots allowed it to catch issues that the other text-only models missed.
Strengths: Visual Analysis and Real-World Context
Gemini’s biggest advantage is its ability to “see” a webpage. You can give it a screenshot, and it will compare the visual layout to the underlying code structure. This helps it spot mismatches that other tools miss. For example, it might notice that something styled to look like a heading isn’t actually using a heading tag, which is a major problem for screen reader navigation.
Its image analysis also extends to generating very detailed and accurate alt text. In our tests, Gemini was the best at describing what was in an image, including the setting, the people, and the actions taking place. This connection to Google’s vast dataset gives it a better understanding of real-world objects and scenes.
Limitations: Generic Guidance
The flip side of Gemini’s strengths is that its advice can sometimes be too generic. While it can accurately describe an image, its guidance on how to write effective alt text that serves a purpose within the page is often vague. Furthermore, its reliance on visual analysis means it can be tripped up by problems that aren’t visible. It can’t experience a keyboard trap or a confusing focus order just by looking at a static image. It provides a good simulation of what a sighted user sees, but it can’t replicate the experience of a non-sighted user.
Expanding the Test: Where ALL AI Tools Fall Short
While each AI has its unique strengths, our tests revealed several critical areas where they all fail. These aren’t minor flaws; they are fundamental gaps in understanding that underscore why human testers remain absolutely necessary for any serious accessibility effort.
The Keyboard Navigation Maze
One of the most important tests for accessibility is unplugging your mouse and trying to navigate a website using only your keyboard. For many users with motor disabilities, this is their primary way of interacting with the web. All three AI models failed to reliably identify keyboard navigation issues. They can’t tell if the focus order is logical, meaning a user might be tabbed to the bottom of the page when they expect to go to the next field. More importantly, they can’t detect “keyboard traps,” where a user can tab into a component (like a video player) but can’t tab back out without reloading the page. Only a human can experience that frustration.
The Cognitive Load Blind Spot
Accessibility isn’t just for users with sensory or motor disabilities. It’s also for people with cognitive and learning disabilities. Is the language on the site clear and simple? The layout overwhelming? Is the checkout process broken into manageable steps? AI is completely blind to these issues. It can’t measure cognitive load or tell you if your information architecture is confusing. It might verify that all your headings are in the correct technical order, but it can’t tell you if the content under those headings is a jumbled mess.
Dynamic Content and Single-Page Apps Modern websites are dynamic. Content frequently changes on the page without a full reload. Think of a social media feed that loads new posts as you scroll or a shopping site where product filters update the results in real-time. Automated AI tools are notoriously bad at testing these experiences. They typically only scan the initial state of the page and miss issues that appear after a user interacts with it. A human tester, on the other hand, moves through the site just like a real user, interacting with dynamic components and finding the bugs that AI scanners can’t see.
The Winning Approach: An AI + Human Workflow
The debate over AI versus human testing is a false choice. The most effective and cost-efficient accessibility strategies don’t pick one over the other; they combine them. A hybrid approach uses automation for what it does best, speed and scale, while relying on human expertise for what machines can’t do: understanding context, logic, and usability.
Step 1: Automate the Obvious with AI
The best place to start is by “shifting left” and integrating automated accessibility testing tools early in the development process. Use an AI-powered scanner, like the one offered by Accessibility-Test.org , to run quick checks on every code change. This provides instant feedback to developers, allowing them to catch that initial 57% of machine-testable issues before they become bigger problems. It’s the most cost-effective way to handle the basics and establish a solid foundation for ADA compliance.
Step 2: Bring in the Humans for What Matters
Once you’ve cleared the automated checks, it’s time for manual testing. This is where you find the critical 43% of issues that AI misses. This process must involve human experts, particularly those who regularly use assistive technologies like screen readers. They will navigate your site’s complex user flows, test your interactive features, and assess the overall experience from the perspective of a person with a disability. This step is not optional; it is the only way to know if your website is truly usable.
Step 3: Creating a Culture of Accessibility
This hybrid workflow is more than just a process, it’s a change in culture. It means making accessibility a shared responsibility for the entire team, not just a final checkbox for the compliance department. It involves providing accessibility training for developers and designers, ensuring they understand the human impact of their work. When your team starts thinking about users with disabilities from the beginning of a project, accessibility stops being a burden and becomes a source of innovation.
What’s Next? The Future of AI in Accessibility
The field of AI accessibility testing is moving fast, and we can expect some big changes over the next couple of years. While AI will never fully replace human testers, its role will become even more powerful.
Smarter AI, Higher Accuracy
First, the accuracy of AI tools will continue to improve. Projections suggest that automated detection rates could climb towards 70% by the end of 2025 or early 2026. This will be driven by better machine learning models trained on enormous datasets of both accessible and inaccessible code, which will help them recognize more complex patterns and reduce false positives.
From Finding to Fixing
The next big shift will be from AI simply finding problems to providing smart solutions. Instead of just flagging an error, future AI tools might offer several code-fix suggestions, complete with explanations of the pros and cons of each approach. Imagine an AI that not only tells you your color contrast is too low but also suggests three alternative, on-brand color pairings that meet WCAG standards.
The Road to WCAG 3.0 Finally, as web standards themselves evolve, so will our testing methods. The upcoming WCAG 3.0 is expected to focus less on rigid technical rules and more on actual user outcomes. This will force a change in how we test for accessibility, and AI will be a central part of that evolution. The tools will need to get better at simulating user experiences to provide meaningful feedback in this new context.
Using Automated Tools for Quick Insights (Accessibility-Test.org Scanner)
Automated testing tools provide a fast way to identify many common accessibility issues. They can quickly scan your website and point out problems that might be difficult for people with disabilities to overcome.
Visit Our Tools Comparison Page!
Run a FREE scan to check compliance and get recommendations to reduce risks of lawsuits
Final Thoughts
Have you considered how many accessibility barriers on your site might be invisible to automated tools? The only way to find out is to look. Start by running a free scan with our accessibility checker, and when you’re ready to find the issues that truly matter, let our human experts show you the way.
Run a Free Scan to Find E-commerce Accessibility Barriers
Want More Help?
Try our free website accessibility scanner to identify heading structure issues and other accessibility problems on your site. Our tool provides clear recommendations for fixes that can be implemented quickly.
Join our community of developers committed to accessibility. Share your experiences, ask questions, and learn from others who are working to make the web more accessible.