Use this when you're evaluating a SaaS vendor, API provider, or third-party integration and need to assess their security posture -- before signing a contract, during a periodic review, or after a vendor has disclosed a security incident.
Related skills: Use
/compliance-mapfor framework knowledge when evaluating vendor certifications. Use/decision-brieffor vendor selection decisions. Use/vendor-evaluationfor broader vendor evaluation beyond security. Use/build-vs-buywhen vendor selection is part of a build-vs-buy decision.
Process
Step 1: Gather context
Collect from the user:
- Vendor name and what service they provide
- What data the vendor will access, process, or store
- Data sensitivity classification -- PII, PHI, financial, proprietary, public
- Integration points with your systems -- API, SSO, data sync, webhook, file transfer
- Contract timeline -- new procurement, renewal, or periodic review
- Any known concerns or prior incidents with this vendor
The depth of assessment should be proportional to risk. A logging SaaS that ingests application metrics gets a lighter review than a payment processor handling credit card data.
Step 2: Generate tailored questionnaire
Based on vendor type and data sensitivity, generate questions across these domains:
For all vendors:
- Data handling: Where is data stored? Is it encrypted at rest and in transit? What encryption algorithms and key lengths?
- Access controls: How is access to customer data controlled? Is there role-based access? How are privileged accounts managed?
- Incident response: What is the vendor's breach notification timeline? Do they have a documented incident response plan?
- Business continuity: What is the RTO/RPO? Where are backups stored? Has DR been tested?
- Compliance: What certifications does the vendor hold (SOC 2, ISO 27001, etc.)? When was the last audit? Can they share the report?
For vendors handling sensitive data (add):
- Subprocessor management: Who are the subprocessors? How are they vetted?
- Data residency: In which regions is data stored? Can data residency be contractually guaranteed?
- Vulnerability management: What is the patching cadence? Is there a bug bounty program?
- Employee security: Background checks? Security training frequency? Least-privilege access enforcement?
- Data lifecycle: How is data deleted on contract termination? Is deletion certified?
For infrastructure/platform vendors (add):
- Network security: Network segmentation? DDoS protection? WAF?
- Logging and monitoring: What audit logs are available to customers? Retention period?
- Multi-tenancy isolation: How is customer data isolated? Shared or dedicated infrastructure?
For vendors with AI/ML capabilities (add):
- AI training data: Is customer data used to train or improve models? Can customers opt out? Is training data shared across customers?
- AI failure transparency: Does the vendor document known failure modes, false positive/negative rates, and confidence thresholds? Can the customer tune sensitivity?
- Baseline requirements: What telemetry quality, log coverage, and tag consistency does the vendor need from the customer's environment to build accurate AI baselines (e.g., for UEBA or anomaly detection)?
- AI explainability: When the AI flags an anomaly or makes a recommendation, can it trace back to specific events, identities, and resources -- or is it a black-box score?
- Model update cadence: How often do detection or analysis models update? Are updates tested against customer-specific baselines before deployment? Can the customer pin a model version?
- AI-specific incident handling: If the AI produces a false positive that triggers an incident response, what's the vendor's process for tuning and preventing recurrence?
Questionnaire length should be proportional to risk. Don't send 200 questions to a low-risk vendor.
Step 3: Define scoring criteria
Build a weighted evaluation with clear pass/fail thresholds:
| Domain | Weight | Pass/Fail Items | Scored Items |
|---|---|---|---|
| Data Encryption | {{weight}}% | Encryption at rest (AES-256 or equivalent) -- MUST PASS | Key management maturity |
| Access Controls | {{weight}}% | MFA for admin access -- MUST PASS | RBAC granularity, access review cadence |
| Incident Response | {{weight}}% | Documented IR plan -- MUST PASS | Notification timeline, testing frequency |
| Compliance | {{weight}}% | SOC 2 Type II or equivalent -- MUST PASS (for sensitive data) | Audit recency, scope coverage |
| Business Continuity | {{weight}}% | None | RTO/RPO, DR testing frequency |
Pass/fail items are non-negotiable. If a vendor fails any pass/fail item, that's a disqualifier unless compensating controls exist.
Step 4: Score vendor responses
Evaluate each response with a risk-rated assessment:
| Domain | Question | Vendor Response | Rating | Notes |
|---|---|---|---|---|
| {{domain}} | {{question}} | {{response_summary}} | Red / Yellow / Green | {{risk_note}} |
Rating definitions:
- Red flag -- immediate disqualifier or critical gap. Example: no encryption at rest for PII, no incident response plan, SOC 2 report expired 18+ months ago.
- Yellow flag -- acceptable with conditions. Example: SOC 2 Type I only (not Type II), patching cadence is quarterly rather than monthly, no dedicated security team but outsourced to reputable firm.
- Green -- meets or exceeds requirements.
Distinguish between what a vendor claims and what they can evidence. "We encrypt everything" without specifying algorithms, key management, or providing a SOC 2 report is a yellow flag.
Step 5: Generate risk assessment
Produce an overall risk profile:
## Vendor Risk Assessment -- {{vendor_name}} -- {{date}}
**Overall Risk Rating:** Low / Medium / High / Critical
**Red Flags:** {{count}}
{{list_red_flags}}
**Yellow Flags:** {{count}}
{{list_yellow_flags}}
**Residual Risks:**
- {{risk_description}} -- Likelihood: {{L}}, Impact: {{I}}
**Compensating Controls (your organization should implement):**
- {{control_description}} -- addresses {{residual_risk}}
Step 6: Produce recommendation
One of three outcomes:
- Approve -- vendor meets security requirements. Proceed with procurement.
- Approve with conditions -- vendor is acceptable if specific conditions are met. List each condition, the timeline for the vendor to meet it, and monitoring requirements.
- Reject -- vendor has critical gaps that cannot be mitigated. Document the specific disqualifiers.
For conditional approval, define:
- Reassessment triggers (vendor incident, contract renewal, material change in data handling)
- Monitoring cadence (quarterly check-ins, annual reassessment)
- Conditions the vendor must meet and by when
Step 7: Generate contract security clauses
Based on findings, draft security-specific contract terms:
## Recommended Contract Clauses
### Data Processing Agreement
- Vendor shall process data only for the purposes specified in {{agreement}}
- Data residency: {{region_requirements}}
### Breach Notification
- Vendor shall notify Customer within {{hours}} hours of discovering a security incident
- Notification shall include: nature of the breach, data affected, remediation steps taken
### Audit Rights
- Customer may audit vendor's security controls {{frequency}} or upon reasonable notice
- Vendor shall provide updated SOC 2 reports within {{days}} days of issuance
### Security SLAs
- Uptime: {{percentage}}
- Critical vulnerability patching: within {{hours}} hours
- Incident response time: {{hours}} hours
### Data Deletion
- Upon contract termination, vendor shall delete all Customer data within {{days}} days
- Vendor shall provide written certification of deletion
Contract clauses should be specific and enforceable. "Vendor will maintain reasonable security" is not a useful clause.
Step 8: Review
Ask the user:
- Are there industry-specific requirements we should add (HIPAA BAA, PCI SAQ)?
- Does legal need to review the contract clauses before they go to the vendor?
- What's the reassessment cadence -- annual, semi-annual, or triggered by vendor incidents?
- Are there existing vendor management policies this assessment should align with?
- Is there a vendor risk register where this assessment should be recorded?
Output location
Deliver as a markdown document. Suggested filename: vendor-assessment-{{vendor-name}}-{{date}}.md. The recommendation section should be extractable for procurement review.
Example Output
Input
- Vendor name and service: Mosaic Analytics — cloud-based HR analytics and workforce planning SaaS; integrates with Workday and ADP to surface headcount, attrition, and compensation benchmarking dashboards
- Data accessed: Full employee records including names, job titles, salary bands, performance ratings, demographic data (age, gender, ethnicity), and Social Security Numbers for payroll reconciliation — classified PII/sensitive HR data
- Integration points: Bidirectional API sync with Workday (nightly), SSO via Okta, webhook callbacks for real-time attrition alerts, file transfer (SFTP) for monthly ADP payroll exports
- Contract timeline: New procurement; legal has a 45-day window before the planned go-live for Meridian Financial Group's Q3 HR transformation program
- Known concerns: Mosaic disclosed a data exposure incident 11 months ago involving misconfigured S3 buckets affecting 3 customers; they claim remediation is complete but no third-party validation was shared
Output (abbreviated)
Vendor Security Assessment — Mosaic Analytics — 2025-07-14
Prepared for: Meridian Financial Group — HR Technology Procurement Assessed by: Information Security & Vendor Risk Assessment type: New procurement — high sensitivity data
Executive Summary
Overall Risk Rating: HIGH
Mosaic Analytics handles some of Meridian's most sensitive employee data, including SSNs, compensation, and protected demographic attributes. The vendor's prior S3 misconfiguration incident — unvalidated by a third party — is a material concern in this context. The assessment identifies 3 red flags and 5 yellow flags. Conditional approval is possible if Mosaic meets specific remediation requirements within 30 days; outright approval is not warranted at this time.
Tailored Security Questionnaire
Data Handling
- Where is employee data stored, and in which AWS regions? Can data residency be contractually locked to US-East/US-West only?
- What encryption algorithm and key length is used for data at rest? Who manages encryption keys — Mosaic, AWS KMS, or a customer-managed key option?
- Following the 2024 S3 misconfiguration incident, what technical controls now prevent public bucket exposure? Has this been independently validated?
- Is SSN data tokenized or masked in analytics dashboards, or is it stored and displayed in plaintext?
Access Controls
- How is Mosaic employee access to customer data controlled? Is access granted per-tenant, and is it logged?
- What MFA requirements apply to Mosaic engineers accessing production systems?
- How often are privileged access rights reviewed and recertified?
Incident Response
- What is your contractual breach notification timeline? Your 2024 incident — how long elapsed between discovery and customer notification?
- Can you share your current incident response plan (redacted acceptable)?
Subprocessor Management
- Which subprocessors process Meridian data (e.g., analytics engines, email providers, support tooling)? How are they assessed?
- Does any subprocessor receive SSN or demographic data?
AI/ML Capabilities
- Mosaic's attrition prediction model — is it trained on data pooled across customers, or per-tenant? Can Meridian opt out of contributing training data?
- When the model flags an attrition risk, can it trace back to specific data inputs, or is it a composite score only?
- How are model updates tested before deployment? Can Meridian pin to a previous model version if an update degrades accuracy?
Data Lifecycle
- Upon contract termination, what is the deletion timeline and method? Is written certification of deletion provided?
Scoring Criteria
| Domain | Weight | Pass/Fail Items | Scored Items |
|---|---|---|---|
| Data Encryption | 25% | Encryption at rest (AES-256) — MUST PASS; SSN not in plaintext — MUST PASS | Key management maturity, customer-managed key option |
| Access Controls | 20% | MFA for all production access — MUST PASS | RBAC granularity, access review cadence |
| Incident Response | 20% | Documented IR plan — MUST PASS; notification ≤ 72 hrs — MUST PASS | Notification SLA, post-incident validation |
| Compliance | 20% | SOC 2 Type II current (< 12 months) — MUST PASS | Scope covers HR data workloads, penetration test recency |
| Subprocessors | 10% | No unvetted subprocessors with SSN access | Subprocessor security evidence |
| Business Continuity | 5% | None | RTO/RPO, DR test evidence |
Vendor Response Scoring
| Domain | Question | Vendor Response | Rating | Notes |
|---|---|---|---|---|
| Data Encryption | Encryption at rest algorithm | AES-256 via AWS KMS; key rotation annual | 🟢 Green | Meets requirement; customer-managed keys available on Enterprise tier |
| Data Encryption | SSN storage and display | SSNs stored encrypted; displayed in full in payroll reconciliation exports | 🔴 Red | SSNs in plaintext in SFTP exports — no masking or tokenization |
| Access Controls | MFA for production access | MFA required for all engineers via Okta | 🟢 Green | Evidenced in SOC 2 report |
| Access Controls | Privileged access review cadence | Quarterly access reviews | 🟡 Yellow | Quarterly is acceptable; semi-annual would be preferred given incident history |
| Incident Response | Breach notification timeline | 72-hour notification per contract | 🟡 Yellow | 2024 incident: 6-day lag between discovery and notification — inconsistent with stated SLA |
| Incident Response | IR plan documentation | Shared redacted IR plan | 🟢 Green | Plan is documented; tabletop exercise completed Q1 2025 |
| Compliance | SOC 2 Type II | SOC 2 Type II issued March 2025; covers core platform | 🟡 Yellow | Report does not explicitly cover SFTP/ADP integration pipeline — scope gap |
| Compliance | Post-incident third-party validation | "Internal audit confirmed remediation" | 🔴 Red | No third-party penetration test or configuration audit post-incident |
| Subprocessors | Subprocessors with SSN access | Fivetran (data pipeline), Snowflake (analytics warehouse), Zendesk (support) | 🟡 Yellow | Zendesk access to support tickets — confirm whether ticket data includes PII |
| AI/ML | Attrition model training data | Cross-customer training with anonymization; opt-out not available | 🔴 Red | Meridian employee data contributes to shared model; no opt-out — unacceptable for demographic data |
| AI/ML | Model explainability | Dashboard shows contributing factors (tenure, performance trend, manager score) | 🟢 Green | Factor-level attribution available; not black-box |
| Data Lifecycle | Deletion on termination | 30-day deletion; certification provided | 🟢 Green | Acceptable |
Risk Assessment
Overall Risk Rating: HIGH
Red Flags: 3
- SSN plaintext in SFTP exports — Payroll reconciliation files transferred to ADP contain unmasked SSNs with no tokenization. Exposure via compromised SFTP credential would give direct access to ~4,200 employee SSNs.
- No third-party validation of 2024 S3 incident remediation — Mosaic's self-attestation is insufficient given that the original misconfiguration involved the same infrastructure now proposed for Meridian data.
- Cross-customer AI training with no opt-out — Meridian's employee demographic data (age, gender, ethnicity) would be used to train models available to Mosaic's full customer base. This creates regulatory exposure under state biometric/demographic data laws and reputational risk.
Yellow Flags: 5
- SOC 2 scope excludes SFTP/ADP pipeline
- 2024 breach notification lag (6 days vs. 72-hour stated SLA)
- Zendesk subprocessor may receive PII in support tickets — unconfirmed
- Quarterly privileged access review cadence (prefer semi-annual or monthly given incident history)
- No customer-managed key option on current contract tier (requires Enterprise upgrade)
Residual Risks
| Risk | Likelihood | Impact |
|---|