Key findings
Read each result with its sample. This is an observational analysis of a small, non-random dataset. The full study contains 10 audits and 8 ACRs; the manual-versus-automated comparison uses only the 7 web audits that included scanning. These figures describe recorded findings, not every barrier that existed or everything a tool could detect.
Read the methodology and limitationsEach figure pools at least 3 clients and identifies its numerator, denominator, and sample size. The web, mobile, criterion-level, and ACR results use different subsets.
- 10 of 10audits recorded screen-reader and contrast issueskeyboard: 9 of 10An audit counts once when it has at least one recorded issue in that test type; this is not a market-wide failure rate.Link to this findingView evidence
- 73.2%of audit–criterion failures had no automated finding recorded41 of 56; n=3 auditsCounts audit–criterion combinations in 3 audits, not individual defects or all criteria an automated tool could evaluate.Link to this findingView evidence
- 192 vs 5Partially Supports vs Does Not Support ratings in ACRsof 409 applicable ratings; n=8 ACRsPooled applicable ratings across 8 ACRs; these labels do not measure issue severity or decide procurement acceptance.Link to this findingView evidence
Issues by test type
All 10 audits, 1,878 issues. High-contrast mode testing found 0 issues.
| Test type | Issues (share) |
|---|---|
| Screen reader | 1,139 (60.6%) |
| Colour contrast | 280 (14.9%) |
| Automated scan | 206 (11.0%) |
| Keyboard | 201 (10.7%) |
| Browser zoom | 37 (2.0%) |
| Other (video, text spacing) | 15 (0.8%) |
Testing with screen readers (JAWS, NVDA, VoiceOver, TalkBack) found more issues than every other test type combined.
Manual vs automated findings on the web
7 web audits, 1,245 issues. Mobile apps are excluded because no automated scan was run on them.
| Recorded test type | Findings | Share (one decimal) |
|---|---|---|
| Manual test types | 1,039 issues | 83.5% |
| Automated scan | 206 issues | 16.5% |
All 7 web audits included an automated scan and recorded findings under automated scanning. Across the pooled web records, 1,039 of 1,245 issues (83.5%) were classified under manual test types and 206 (16.5%) under automated scanning. The per-audit average manual share was 85.4%. In the 3 audits with WCAG-criterion data, 403 of 485 issue records (83.1%) were classified as manual findings.
What this comparison establishes: 1,039 of 1,245 web issue records were assigned to manual test types. It does not establish that automated tools universally miss 83.5% of accessibility issues. The audits were not a controlled comparison of all tools or their maximum possible coverage. Contrast-analyzer checks were classified as manual, audit scopes varied, and the mobile-app audits were excluded because they had no automated scan.
Review the study limitationsWeb platforms vs mobile apps
Share of each platform's issues by test type. Web: 7 audits, 1,245 issues. Mobile: 3 apps, 633 issues.
On mobile, screen-reader testing with VoiceOver and TalkBack found 518 of 633 issues. No automated scan category was used for the apps.
Web platforms / Mobile apps: Screen reader · Contrast · Keyboard · Automated · Zoom · Other
Which tests failed in how many audits
| Test type | Audits with at least one issue |
|---|---|
| Screen reader | 10 of 10 |
| Colour contrast | 10 of 10 |
| Keyboard | 9 of 10 |
| Automated scan | 7 of 10 (7 of 7 web) |
| Browser zoom | 6 of 10 (6 of 7 web) |
| Video | 3 of 10 |
| Text spacing | 2 of 10 |
| High-contrast mode | 0 of 10 |
WCAG criteria behind the issues
| WCAG 2.x success criterion | Level | Issues | Share |
|---|---|---|---|
| 1.3.1 Info and Relationships | A | 115 | 23.7% |
| 4.1.2 Name, Role, Value | A | 63 | 13.0% |
| 1.1.1 Non-text Content | A | 59 | 12.2% |
| 2.1.1 Keyboard | A | 34 | 7.0% |
| 1.4.11 Non-text Contrast | AA | 28 | 5.8% |
| 2.4.3 Focus Order | A | 26 | 5.4% |
| 1.4.3 Contrast (Minimum) | AA | 25 | 5.2% |
| 3.3.2 Labels or Instructions | A | 13 | 2.7% |
| 4.1.3 Status Messages | AA | 12 | 2.5% |
| 1.4.4 Resize Text | AA | 9 | 1.9% |
| 1.4.10 Reflow | AA | 7 | 1.4% |
From the 3 audits with WCAG-criterion-level data only (485 issues). Smaller sample; read as indicative.
The table lists the 11 criteria failed in all 3 audits; together they hold 391 of 485 issues (80.6%).
The top 3 (1.3.1, 4.1.2, 1.1.1) hold 237 of 485 issues (48.9%).
Level A, the baseline, holds 358 of 483 mapped issues (74.1%).
Criteria from WCAG 2.0 hold 419 of 483 (86.7%) and WCAG 2.1 additions hold 64 (13.3%). Criteria new in WCAG 2.2 hold 0, although all 3 audits tested against 2.2.
Of 56 audit-criterion failures, 41 (73.2%) had no automated finding at all. Issues under criteria that automation never flagged in the same audit total 209 of 485 (43.1%).
How ACRs rated products
8 ACRs (VPAT-based), one per product. Shares are of applicable ratings; 197 further ratings were Not Applicable.
| Conformance rating | Ratings | Share (rounded) |
|---|---|---|
| Supports | 212 ratings | 52% |
| Partially Supports | 192 ratings | 47% |
| Does Not Support | 5 ratings | 1% |
Across 8 ACRs, 46.9% of applicable ratings were Partially Supports and 1.2% were Does Not Support. In 6 of the 8 ACRs, Does Not Support was never used, even though every product's audit logged dozens to hundreds of issues. Not Applicable made up 197 of 606 ratings (32.5%). For buyers, a Partially Supports rating is where the real issues are recorded.
Issue counts in this dataset
- Total issues, all audits
- 1,878 (n=10)
- Median issues per audit
- 174.5 (n=10)
- Median issues per page or screen
- 5.4 (n=8)
What it means for vendors and buyers
For vendors
Use automated scanning as one part of an evaluation. In this dataset, 16.5% of recorded web findings were classified under automated scanning; the distribution is not a universal detection rate.
Test with real assistive technology on every platform you ship, including VoiceOver and TalkBack for apps.
Give foundational barriers careful attention. In the 3-audit criterion-level subset, structure, names and roles, and text alternatives accounted for 237 of 485 findings (48.9%). Prioritize corrections by user impact and critical workflows as well as the applicable criteria.
Write ACR remarks that explain each Partially Supports rating; buyers read them closely.
For procurement teams
Ask for the audit behind an ACR, including whether manual screen-reader testing was done.
Read every Partially Supports remark: in this sample, that is where most known issues sit.
Treat an ACR with no Does Not Support ratings as a prompt for questions, not proof of conformance.
Put WCAG 2.1 AA, remediation timelines, and retesting into contract terms.
To apply these findings to your testing plan, use our guide to manual and automated accessibility testing for practical advice on workflows, keyboard checks, and screen-reader evaluation.
Compliance dates (WCAG 2.1 Level A and AA)
| Rule | Who | Date |
|---|---|---|
| ADA Title II (DOJ) | State and local governments, population 50,000+ | April 26, 2027 |
| ADA Title II (DOJ) | Under 50,000, and special district governments | April 26, 2028 |
| Section 504 (HHS) | HHS funding recipients, 15+ employees | May 11, 2027 |
| Section 504 (HHS) | HHS funding recipients, fewer than 15 employees | May 10, 2028 |
Both rules cover web content and mobile apps an entity provides directly or through contractual, licensing, or other arrangements, which brings vendor products into scope. Both dates were extended by one year in 2026.
Methodology and limitations
What was analyzed
The research dataset includes:
- 10 manual accessibility audits of 10 distinct products: 7 web platforms and 3 mobile apps (iOS and Android).
- 8 Accessibility Conformance Reports (VPAT-based), one per product.
- Audit findings covering WCAG 2.x Level A and AA, with the criterion-level subset mapped to WCAG 2.2.
The underlying audits and ACRs were included for aggregate research analysis. This report does not identify individual clients, products, organizations, sectors, or engagements.
Test types: screen reader, keyboard, browser zoom, colour contrast, high-contrast mode, text spacing, video, and automated scanning on web.
Assistive technology: JAWS and NVDA on Windows, VoiceOver on macOS and iOS, and TalkBack on Android, across major browsers.
How the numbers were combined
Issue counts were summed across audits, so larger audits weigh more. Where it matters, the per-audit average is also given.
Test-type figures use all 10 audits. WCAG-criterion figures use the 3 audits with issue-level criterion data and are labelled that way.
ACR shares are of applicable ratings (Supports, Partially Supports, Does Not Support). Products tested on two platforms count once.
Every published figure pools data from at least 3 clients. No individual client, product, engagement, sector, audit, or ACR is identified.
Limitations
The sample is small and non-random. It reflects accessibility audits and ACRs available within the research dataset rather than a statistically representative sample of all websites, applications, or software products.
The test-type distribution describes the method recorded for each finding. It is not a controlled study of automated tools, a measurement of every defect present, or an estimate of the maximum coverage automation could achieve.
Criterion-level results rest on 3 audits.
Severity is not compared, because severity scales were not consistent across audits.
Audit scope, testing methodology, product complexity, and issue-grouping practices can affect the number and type of findings recorded.
ACRs covered different row sets (WCAG only vs WCAG plus other standards), so only pooled shares of applicable ratings are reported.
Colour contrast was checked with a contrast-analyzer tool and is counted as a manual test type.
These findings should therefore be interpreted as observations from this dataset, not estimates of accessibility performance across the broader market.
Appendix: Methodology
Test types
| Test type | What it checks | Main WCAG criteria |
|---|---|---|
| Screen reader | A tester navigates and operates the product with a screen reader. The check is that content, structure, controls, and their names, roles, and states are announced correctly. | 1.3.1, 4.1.2, 1.1.1, 2.1.1, 2.4.4, 3.3.2 |
| Colour contrast | Measures the contrast of text, and of interface components and graphics, against their backgrounds, using a contrast-analyzer tool. | 1.4.3, 1.4.11 |
| Keyboard | Operates everything with the keyboard alone. Checks that focus is visible, moves in a logical order, and reaches every control. | 2.4.7, 2.4.3, 2.1.1 |
| Automated scan | A rules-based scan of page code with axe (web only). | 1.3.1, 1.1.1, 4.1.2 |
| Browser zoom | Enlarges content in the browser. Checks that text stays readable and the layout reflows without loss of content or function. | 1.4.4, 1.4.10 |
| Text spacing | Increases line, paragraph, letter, and word spacing. Checks that no content is clipped or overlaps. | 1.4.12 |
| Video / media | Checks that prerecorded media has captions and an audio description or media alternative. | 1.2.2, 1.2.3 |
| High-contrast mode | Turns on the operating system's high-contrast mode. Checks that content and controls stay visible and usable. | No issues logged in any audit |
Main WCAG criteria are the success criteria under which each test type's issues were most often logged in the 3 audits with criterion-level data.
Assistive technology and platforms
| Platform | Assistive technology and browser |
|---|---|
| Windows 10 | JAWS and NVDA with Chrome; NVDA with Firefox; JAWS with Edge |
| macOS | VoiceOver with Safari |
| iOS | VoiceOver (with Safari for web content) |
| Android | TalkBack (with Chrome for web content) |
| Tools | axe (automated scan, web) and Colour Contrast Analyzer |
As recorded in the audit files. Scope varied by product, so not every audit used every combination.
How issues were logged and counted
Each issue is one record with: test type, method (manual or automated), WCAG success criterion, severity, environment, page or screen, steps to reproduce, actual and expected result, user impact, and a recommended fix.
Repeated instances of the same problem can be grouped in one issue record, with the instances noted. Counts in this report are issue records, not instances.
For the 3 audits with issue-level records, test-type and criterion counts come straight from those records.
For the other 7 audits, test-type counts come from each audit's totals by test type.
For mobile apps tested on both iOS and Android, issues on each platform are counted separately and then summed.
Manual means every test type other than the automated scan. Colour contrast counts as manual, following the audit template.
For criterion figures, each issue is assigned to the WCAG success criterion it was logged under. Issues logged without a criterion are excluded from criterion shares.
How ACR ratings were tallied
Each ACR uses the VPAT conformance terms Supports, Partially Supports, Does Not Support, and Not Applicable. Not Evaluated is reserved for AAA criteria and does not appear.
Ratings were counted per report, then summed across the 8 ACRs.
Shares are of applicable ratings (Supports plus Partially Supports plus Does Not Support); Not Applicable is reported separately. This is because the set of rows differed between reports.
Products rated separately for iOS and Android are counted once, using the iOS ratings.
Anonymity and aggregation
Every published figure pools data from at least 3 clients.
No individual client, product, engagement, sector, audit, or ACR is identified. Per-audit figures, exact engagement dates, and other details that could reasonably identify an underlying organization are not published.
Results appearing in fewer than 3 clients are included only within appropriately aggregated figures.
About this report
- Published by
- CompliCorp
- Research and analysis
- CompliCorp
- What CompliCorp does
- Manual accessibility testing, WCAG audits, and VPAT/ACR reports
- Website
- https://complicorp.com
- Author
- David LoPresti, Founder & CEO, CompliCorp
This report presents anonymous, aggregated analysis of accessibility audit and ACR data included in CompliCorp’s research dataset. It does not identify or describe individual client engagements.
Suggested citation
LoPresti, David. “Beyond the Scan: What 1,878 Accessibility Issues Say About Manual Testing.” CompliCorp, 2026.
https://complicorp.com/research/manual-vs-automated-accessibility-testing
When citing a result, include its numerator, denominator, sample size, and recorded test category. The direct links beside the key findings identify the relevant result; keep the scope and limitations with the claim.