We publish this knowing buyers can use it to evaluate Alacrinet too. That is the point. If a vendor cannot answer these in writing, the proposal is not finished.
By Bailey Besheer ·
On a sales call, most pentest vendors sound interchangeable. The work they deliver is not. The price gap between them runs into six figures, and so does the quality gap. This checklist gives you a way to tell them apart before you sign anything.
Use it on any vendor. Including us.
File · How to
Send the twelve questions to every vendor on your shortlist. Ask for written answers, not verbal ones on a sales call. If a vendor will not commit in writing, treat the silence as an answer.
Score each answer against the Strong Answer and Red Flag patterns under each question. The pattern that emerges across twelve answers will tell you more about the vendor than any pitch deck.
Download the PDF version to send around internally or attach to your RFP.
File · What to
The twelve questions below are written so real pentest shops can answer them specifically and confidently. Commodity vendors will dodge, deflect to a sales engineer, or give you something vague. Two things to watch for in every answer:
File · We don't
This checklist is the only thing we'll send you. No drip campaign. No 'checking in' emails. No sales rep assigned to your account. If you want to talk to us, contact details are at the bottom. If you don't, you'll never hear from us.
Operator Note
File · Question 1
Why it matters. These are two different metrics that commodity vendors love to blur together. Manual validation is the bar for findings: every CVE, every misconfiguration, every credential in the report should have been confirmed by a human before it shipped. Manual exploitation is the bar for testing: how much of the actual attack-chain work was done by an operator instead of an automated scanner.
Strong answer. 100 percent of findings manually validated before reporting. Manual exploitation typically 60 to 90 percent depending on engagement scope, with some discovery and reconnaissance appropriately tooled. The vendor can articulate exactly what gets tooling, what gets validated by hand, and why.
Red flag. Findings list contains thousands of unvalidated CVEs. Vendor can't distinguish between scanner-discovered, scanner-validated, and manually-exploited findings. "We use a combination of approaches." "It depends on the engagement."
File · Question 2
Why it matters. PTES, NIST 800-115, and OSSTMM are the recognized methodology standards in this space. Mapping findings to MITRE ATT&CK is the minimum bar for a credible deliverable. A shop that can't name a framework is improvising.
Strong answer. Names a specific framework (typically PTES) and can describe how findings are mapped to ATT&CK techniques.
Red flag. "We follow industry best practices." "Our methodology is proprietary." No working familiarity with ATT&CK.
File · Question 3
Why it matters. Real pentest shops use scanners selectively for reconnaissance and validate every finding manually. Commodity shops include raw scanner output in deliverables, which inflates finding counts with low-quality false positives.
Strong answer. A specific position on scanner use. Real shops do not deliver raw scanner output as findings. They may use Nmap for port discovery and exclude vulnerability scanners (Nessus, Qualys, Rapid7) from final reporting.
Red flag. Findings list contains thousands of CVEs from scanner output. Unable to articulate the line between scanner findings and manually validated findings.
Operator Note
File · Question 4
Why it matters. Commodity engagements are often staffed by offshore or junior testers with one senior reviewer. A US-based, all-senior team carries different cost and meaningfully different output quality.
Strong answer. Specific location of testers (US-based, EU-based), specific seniority distribution, and willingness to disclose who will actually be on the engagement before contract signing.
Red flag. "We have a global delivery team." Unwilling to specify locations. "We'll assign appropriate resources."
File · Question 5
Why it matters. OSCP, OSEP, and OSCE3 are hands-on testing certifications from Offensive Security. CRTO and CRTL are red-team certifications from Zero-Point Security. IAT III is the DoD operator-level cyber certification, requiring hands-on practical demonstration. CSIS is a hands-on operator credential. These are the credentials that map to actual pentest work. CISSP and Security+ alone are governance and management certs. They're useful, but they're not pentest credentials by themselves. The team's cert mix tells you whether you're hiring operators or auditors.
Strong answer. Names specific hands-on operator certifications (OffSec, SANS GIAC hands-on, Zero-Point, DoD IAT III, CSIS), with depth across the team.
Red flag. Only governance certs (CISSP, CISM, Security+) with no hands-on credentials behind them. Vague references to "industry certifications." Certifications listed are sales-focused or management-oriented.
Operator Note
File · Question 6
Why it matters. This question separates real shops from commodity shops more cleanly than anything else on the list. A real pentest report template demonstrates attack-narrative structure, exploitation-evidence formatting, chained attack-path documentation, and prioritized remediation guidance. A commodity template is a CVSS-sorted list of CVEs with vendor patch advisories pasted in. You do not need to see a past client's data to evaluate the structure. A vendor handing you a redacted past client deliverable is also telling you something about their discretion practices, which is what Question 12 covers.
Strong answer. Yes. Vendor provides a synthetic or sanitized template that demonstrates attack narratives, chained paths, exploitation-evidence formatting, and remediation guidance structure. No client data involved.
Red flag. "We can't share samples for confidentiality reasons," with no alternative offered. Sample is a generic vulnerability list with no narrative or exploitation evidence. Or: vendor hands over past client work with names redacted, which signals weak discretion practices on real engagements.
File · Question 7
Why it matters. A vulnerability is a possibility. A demonstrated attack path is a proven impact. Real pentests prioritize the latter; commodity shops can only deliver the former.
Strong answer. Discusses how the report ties individual findings into chained attack paths, with explicit notation of which findings were exploited end-to-end versus identified but not exploited.
Red flag. All findings presented as standalone vulnerabilities. No narrative tying findings together. Unable to articulate the difference between a vulnerability and an attack path.
File · Question 8
Why it matters. Scanner-based deliverables produce thousands of findings, most of them informational or low-severity. Real pentest deliverables produce dozens of curated findings, with a much higher proportion of critical and high entries because each one has been validated.
Strong answer. Specific numbers, given in real time. As a reference point, the enterprise engagements we run land in the dozens of curated findings, not the thousands, with a meaningful share rated critical or high because each one was validated by hand. A vendor who knows their own work can give a ballpark without consulting anyone.
Red flag. "We find as many as are present." Very high finding counts (thousands). Unable to give a rough range without checking delivery.
File · Question 9
Why it matters. This separates work that helps your team remediate from work that just documents the problem.
Strong answer. Each finding includes proof-of-concept content, reproduction commands, screenshots showing exploitation, and remediation guidance specific to your environment.
Red flag. Findings only reference CVE IDs and CVSS scores. Remediation advice is generic ("apply vendor patches"). No proof of exploitation is included.
Operator Note
File · Question 10
Why it matters. A pentest is only valuable if findings get fixed. Most shops cap remediation testing at 30 or 90 days. That's almost never enough time for an internal team to actually work through a large finding list. The retest policy tells you whether the vendor is in the relationship for the deliverable or for the outcome.
Strong answer. Unlimited or very long retest windows. No per-finding cap. Willingness to validate fixes as your team works through them. Past examples of staying engaged beyond standard windows without additional billing.
Red flag. Hard 30-day or 90-day cap. Each retest is a separate billable engagement. No proactive offer of validation support.
File · Question 11
Why it matters. Real pentest engagements regularly surface critical findings that can't wait for the final report. Credentials sitting in cleartext. Active exploitation paths into your environment. Evidence of prior compromise. Good vendors call you the same day. Bad ones bury it in the report a month later.
Strong answer. A defined notification SLA (24 hours is typical), named escalation contacts on both sides, and a real-time channel (Slack, Teams, or signed email) for events that can't wait.
Red flag. "We document it in the report." No SLA. No interim communication mechanism. Critical findings handled the same as informational findings.
File · Question 12
Why it matters. Testing generates the most sensitive material your security program will ever produce: exploitation evidence, captured credentials, working notes, and raw data on what's exploitable in your environment. What the vendor does with that material after the engagement ends is a question of operational discretion, not just legal compliance.
Strong answer. A documented destruction policy with specific timeline (typical: 30 days post-engagement). All raw data, exploitation evidence, captured credentials, and working notes destroyed. No retention for case studies or marketing reuse.
Red flag. Vague answer. "We store it securely." No commitment to a destruction timeline. Vendor publishes case studies or anonymized engagement summaries. Anonymizing rarely holds up: the architecture, the sector, and the timeline together are often enough for the wrong reader to name the client.
File · How we
Our testers are senior, US-based, and do exploitation by hand. We work to PTES methodology and map findings to MITRE ATT&CK. Reports lead with demonstrated attack paths rather than vulnerability lists. Remediation validation is unlimited and uncapped. Critical findings get escalated inside 24 hours rather than buried in a final report. Testing artifacts get destroyed 30 days after engagement end, with no exceptions for case studies or marketing reuse. Clients hand us a level of access almost nothing else in their security program receives, usually at a moment when something has gone wrong or is about to, and that is why we answer every question on this list in writing and never turn the engagement behind it into marketing. We do not publish case studies for the reasons explained on /why-no-case-studies.
File · Get a
Send us the proposal you're evaluating and we'll mark it up against the twelve questions in this checklist. You get a candid technical read in writing within 48 hours. We don't ask for a meeting and we don't follow up unless you ask us to.
Proposal Second Read
SLA · 48H WRITTENFile · FAQ
Q1 Can I download a copy of this checklist?
Yes. The 12 questions on this page are the checklist. Copy them into your RFP, send them to every vendor on your shortlist, and require written answers.
Q2 Will Alacrinet answer these in writing for our RFP?
Yes. Bailey signs and dates the written answers for every prospect engagement. If a competitor will not, that is data.
Q3 Why is the retest question so heavily weighted?
Because the retest is where vendor incentives diverge from buyer incentives. Per-finding retest fees turn remediation into a billable event. Unlimited retest aligns the vendor with the buyer.
Talk to an Operator
Real operators. Real attack paths. Real business impact. Talk to us about your security goals.