ATS Evaluation Checklist: 20 Criteria That Actually Matter
An ATS evaluation checklist is a weighted scoring framework that helps hiring teams compare applicant tracking systems across workflow fit, AI capabilities, pricing transparency, data portability, and vendor reliability — rather than relying on feature lists and demo impressions. This guide covers 20 criteria grouped into six categories, with importance weights, what to look for, and the red flags that should stop a deal.
Demos Are Scripted
Why Feature Lists Lie
Every ATS vendor has a polished demo. They walk you through the happy path — the one scenario where everything works perfectly. But your team does not live on the happy path. You have edge cases, unusual workflows, and integrations that were not part of the script.
Recruitee’s vendor evaluation framework found that 15–25% of total contract value is lost on poor vendor choices. That is not just the subscription cost — it includes implementation time, training hours, data migration pain, and the opportunity cost of a tool your team stops using after three months.
A structured evaluation checklist forces you to ask the right questions before the contract is signed, not after. Below are 20 criteria, grouped into six categories, each with an importance weight so you can score vendors objectively.
Workflow Compatibility
How well the ATS fits the way your team actually hires. Weight: 25% of total score.
1. Pipeline Customization
What to look for: Can you rename stages, add custom stages, create different pipelines for different roles? Can hiring managers drag candidates between stages without admin help?
Red flag: Fixed pipeline stages that cannot be modified, or customization requires support tickets.
2. Permission Model
What to look for: Role-based access control that lets you give hiring managers visibility into their roles without exposing salary data or other teams’ pipelines.
Red flag: All-or-nothing permissions where users either see everything or nothing.
3. Collaborative Evaluation
What to look for: Structured scorecards, shared notes, and the ability for multiple team members to evaluate candidates independently before seeing each other’s feedback.
Red flag: No scorecard system, or evaluations are visible to everyone before submission (anchoring bias).
4. Automation Rules
What to look for: Trigger-based automations — auto-reject below a score threshold, auto-advance past a threshold, send templated emails at stage changes, notify hiring managers when new applicants arrive.
Red flag: Manual-only workflows with no automation layer, or automations locked behind enterprise tiers.
AI & Screening Capabilities
How the system evaluates candidates — learn how AI screening works. This is where most vendors oversell. Weight: 25% of total score.
5. Screening Methodology
What to look for: Does the AI use signal-based scoring that evaluates measurable outcomes, or simple keyword matching that counts term frequency? Ask for a specific explanation of what the algorithm evaluates.
Red flag: Vendor says “AI-powered” but cannot explain the methodology, or screening is purely keyword-based.
6. Bias Handling
What to look for: Documented approach to bias detection and mitigation. Can you audit the AI’s decisions? Does it provide explainability for each score?
Red flag: No bias documentation, no explainability, or the vendor deflects the question.
7. Resume Parsing Accuracy
What to look for: Test with 10 real resumes from your candidate pool. Check whether the parser correctly extracts job titles, dates, skills, and education across different formats (PDF, DOCX, creative layouts).
Red flag: Parser fails on non-standard formatting, or test is not allowed during trial.
8. Candidate Sourcing & Network
What to look for: Built-in sourcing tools or access to a candidate network. Does the system help you find candidates, or only manage the ones who apply?
Red flag: Zero sourcing capability and no integration with sourcing tools.
Integration & Ecosystem
How the ATS connects with your existing tools. Weight: 15% of total score.
9. Calendar & Email Integration
What to look for: Native Google Workspace or Outlook integration for calendar syncing and email tracking. Two-way sync, not just one-way.
Red flag: Requires a third-party connector (Zapier) for basic calendar integration.
10. Communication Tools
What to look for: Native Slack or Teams integration for real-time notifications. Hiring updates should reach your team where they already work.
Red flag: Email-only notifications with no webhook or chat integration.
11. API Access
What to look for: A documented REST API that lets you build custom integrations, pull reporting data, or connect to your HRIS.
Red flag: No API, or API access is only available on enterprise plans.
12. Job Board Distribution
What to look for: One-click posting to major boards (LinkedIn, Indeed, Glassdoor) and the ability to add custom boards. Free distribution channels matter — paid board credits expire and add hidden cost.
Red flag: Board posting requires additional subscription fees beyond the ATS plan.
Pricing Transparency
What it actually costs — see our ATS pricing comparison for benchmarks. Weight: 15% of total score.
13. Pricing Model
What to look for: Flat-rate or per-job pricing that is predictable. Per-seat pricing punishes team growth — adding a hiring manager should not increase your bill by $50/month. Check our pricing page for an example of flat-rate structure.
Red flag: Per-seat pricing, hidden implementation fees, or “contact sales” with no published prices.
14. Free Tier or Trial
What to look for: A genuine free tier (not a 14-day trial) that lets you test real workflows before committing money. CurriculoATS offers a free Starter plan with AI screening and 1 active job.
Red flag: No free trial, or the trial is so limited it is useless for evaluation.
15. Contract Terms
What to look for: Month-to-month options, or annual contracts with a reasonable exit clause. What happens to your data if you cancel mid-contract?
Red flag: Multi-year lock-ins, auto-renewal without notice, or data deletion upon cancellation.
16. Total Cost of Ownership
What to look for: Factor in implementation fees, training costs, per-seat add-ons, and integration costs. A $149/month tool with $5,000 in setup fees and $50/seat add-ons costs far more than the sticker price suggests.
Red flag: Setup fees over $1,000, or mandatory paid training programs.
Support & Onboarding
How fast you get live and what happens when things break. Weight: 10% of total score.
17. Implementation Timeline
What to look for: Startup-friendly ATS platforms should be live in hours or days, not weeks. If implementation requires dedicated project managers and 6-week timelines, the product is built for enterprises, not growing teams.
Red flag: Implementation timelines measured in months, or mandatory professional services engagements.
18. Support Responsiveness
What to look for: Chat or email support with response times under 4 hours during business hours. Check G2 and Capterra reviews specifically for support quality — it is the most common complaint across all ATS platforms.
Red flag: Support only via ticket system with 24-48 hour SLAs, or live support locked behind enterprise plans.
Security & Data
Data portability and compliance readiness. Weight: 10% of total score.
19. Data Export
What to look for: Full data export in standard formats (CSV, JSON). You should be able to export candidates, pipeline data, notes, scorecards, and communication history at any time, without contacting support.
Red flag: No self-service export, export limited to basic fields, or vendor charges for data extraction.
20. Compliance Readiness
What to look for: GDPR compliance documentation, data retention policies, candidate consent management, and SOC 2 certification (or evidence of pursuing it). If you hire internationally, GDPR readiness is non-negotiable.
Red flag: No published privacy documentation, no data retention controls, or unclear candidate consent handling.
How CurriculoATS Scores
An honest assessment against our own checklist. We score well on pricing, AI methodology, and speed — and acknowledge where enterprise platforms have deeper ecosystem coverage.
| Category | Weight | CurriculoATS | Greenhouse | Lever |
|---|---|---|---|---|
| Workflow Compatibility | 25% | Strong | Strong | Strong |
| AI & Screening | 25% | Signal-based | Keyword-based | Keyword-based |
| Integration Ecosystem | 15% | Core covered | 400+ integrations | 300+ integrations |
| Pricing Transparency | 15% | $0–$50/mo flat | $6K+/yr, per-seat | ~$3.5K/yr, custom |
| Support & Onboarding | 10% | Live in minutes | Weeks to implement | Days to implement |
| Security & Data | 10% | Full export, GDPR | SOC 2, GDPR | SOC 2, GDPR |
Running Your Evaluation
Step 1: Longlist. Start with 8–10 vendors. Use our comparison hub and the buyer’s guide to narrow based on team size, budget, and must-have features.
Step 2: Score each vendor. Use the 20 criteria above. Rate each on a 1–5 scale, multiply by the importance weight, and total the scores. This gives you a single comparable number per vendor.
Step 3: Trial the top 3. Never buy based on a demo alone. Run real candidate data through each system. Have your hiring managers use it for a real role. The trial will reveal what the demo hid.
Step 4: Check references. Ask for 2–3 customer references at your company stage. A vendor with 4,000 enterprise customers may not serve your 20-person startup well.
If you want to start with a free trial that requires no commitment, CurriculoATS has a free Starter plan with AI screening and 1 active job.
How many ATS vendors should we evaluate before deciding?
Evaluate 3 to 5 vendors. Fewer than 3 and you lack comparison data. More than 5 and decision fatigue sets in without meaningfully better outcomes. Start with a longlist of 8–10, then narrow to 3–5 for demos and trials.
What is the most important ATS evaluation criterion?
Workflow compatibility. An ATS with every feature in the world is useless if it does not fit how your team actually works. Pipeline customization, permission models, and integration with your existing tools matter more than a long feature list.
Should we require a free trial before purchasing?
Yes. Any vendor that does not offer a free trial or sandbox is hiding something. Demos are scripted — they show the best-case scenario. A trial lets your actual team test real workflows with real data.
How much should a startup expect to pay for an ATS?
Startups under 200 employees should expect to pay between $0 and $150 per month. Some platforms like CurriculoATS offer a free tier with AI screening. Enterprise platforms start at $6,000+/year and scale past $125,000 for large teams.
What ATS red flags should we watch for during evaluation?
The biggest red flags: no free trial, per-seat pricing that punishes growth, inability to export your data, vague answers about AI methodology, annual contracts with no opt-out, and implementation timelines measured in months.
How do we evaluate AI screening capabilities in an ATS?
Ask the vendor to explain exactly what their AI evaluates. Keyword matching counts term frequency and penalizes different terminology. Signal-based scoring evaluates measurable outcomes regardless of the specific words used. The methodology determines whether AI improves hiring quality or just automates a flawed process.