Two approaches to vendor risk assessment dominate the market, and practitioners often treat them as competitors when they work best as complements. Security ratings - from providers such as BitSight, SecurityScorecard, and UpGuard - are outside-in scores derived from observable internet data with no vendor cooperation required. Evidence-based TPRM is the broader practice of collecting, validating, and cross-referencing vendor-provided artifacts: questionnaire responses, audit reports, certifications, and contractual controls, checked against observable facts. Understanding what each actually measures is the starting point for using both well.
What security ratings measure - and how
A security rating is assembled from signals any observer can collect without the vendor's knowledge: exposed ports and services, certificate and DNS configuration, email authentication records, software versions visible from the outside, and leaked credentials found in breach databases. Providers aggregate these into a numerical score using proprietary algorithms. The methodology varies by provider, which is why scores for the same vendor can differ significantly across services.
The genuine strengths of this approach are real. It requires no vendor participation, so a team can screen hundreds of vendors quickly. It updates continuously as the vendor's external posture changes. And it surfaces observable problems - an exposed administrative interface, an expired certificate, a credential in a breach database - that a questionnaire would not catch, because no vendor will voluntarily disclose them.
The transparency and reliability critique
Security ratings have attracted sustained scrutiny on two related grounds: opacity and the limits of outside-in signals.
On opacity: the weighting and methodology behind most ratings scores are proprietary. A vendor disputing its score cannot independently reproduce the calculation, and scores for the same company can vary materially across providers because each weighs signals differently. Buyers relying on a score to support a risk decision need to be able to explain and defend it - to auditors, to regulators, and to the vendor if challenged. NIST SP 800-161 Revision 1, the federal C-SCRM standard, frames supply chain risk management as a formal process with documented, assessable controls; a score whose provenance is opaque is difficult to map to those requirements.
On the limits of outside-in signals: a security rating reflects what the internet sees, not what your data is protected by. A vendor can have a strong external score and still lack basic internal controls over data handling, access management, or incident response - controls that are invisible to an outside-in scan by design. Conversely, a vendor with a minor configuration finding may have excellent compensating controls. The score captures one real and important dimension of vendor risk; it does not substitute for a view of the controls that govern how the vendor handles your data.
What evidence-based TPRM measures
Evidence-based assessment works from the inside out. It collects artifacts the vendor provides - questionnaire responses, SOC 2 Type II reports, ISO 27001 certificates, penetration test summaries, contractual commitments - and validates them against observable facts and the assessor's judgment. The process is more time-intensive and requires vendor cooperation, but it addresses questions ratings cannot:
- How does the vendor control access to your data, and how do they manage privileged accounts?
- What are their incident detection and response procedures, and what are your notification rights?
- Do they encrypt data in transit and at rest, with what key management practices?
- What contractual rights do you have to audit, require remediation, or exit the relationship?
- What sub-processor and subcontractor obligations apply when the vendor further outsources?
Standard questionnaire frameworks such as the Shared Assessments SIG and the CSA CAIQ provide structured, industry-recognized question sets aligned to control frameworks. They give both parties a common language and make responses comparable across vendors and over time.
What regulators and frameworks actually require
The distinction between ratings and evidence-based assessment has direct compliance implications, because most regulatory obligations cannot be satisfied by a security score alone.
DORA (the EU Digital Operational Resilience Act, applicable to financial-sector entities from January 2025) requires specific contractual provisions with ICT third-party providers: audit rights, exit strategies, documented risk assessments, incident notification timelines, and concentration risk management. A security score satisfies none of these provisions on its own.
The European Banking Authority's Guidelines on Outsourcing Arrangements (EBA/GL/2019/02) require documented assessments of sub-outsourcing chains, concentration risk, and the ability to audit or inspect service providers - requirements that point squarely to evidence-based processes.
NIST SP 800-161 r1 frames C-SCRM as a formal program with a supply chain risk management plan, documented controls, and ongoing assessments. A security rating can contribute as a monitoring signal, but it cannot replace the documented, auditable assessment process the framework requires.
Where ratings fit in a compliance context is as a continuous monitoring overlay: a material change in a vendor's external score is a prompt to revisit your evidence-based assessment, not a standalone compliance finding.
A practical tiering framework
The most effective programs do not choose between ratings and evidence - they allocate each method by vendor tier and assessment stage.
| Vendor tier | Primary assessment method | Role of the complementary method |
|---|---|---|
| Critical - high data access, operational dependency, or regulated data | Full evidence-based assessment: SIG or equivalent questionnaire, SOC 2 review, contractual controls, and sub-processor mapping | Security rating as continuous monitoring; a score change triggers re-review |
| Significant - moderate access or standard SaaS with limited data scope | Targeted questionnaire and certification review | Rating for broad triage and change detection between formal reassessments |
| Low - minimal data access or no sensitive processing | Security rating as the primary signal | Evidence requested only if the rating declines materially or an incident is detected |
For how to operationalize this tiering within a broader program, see what is TPRM and continuous vendor monitoring.
Choosing and combining the approaches
Ratings and evidence-based assessment answer different questions and work best together. Use ratings to quickly screen and continuously monitor the external posture of a broad portfolio. Use evidence-based assessment for any vendor where the complete picture matters - critical and regulated data, contractual obligations, and compliance evidence. Platforms that combine both - pairing continuous external monitoring with evidence collection and validation - aim to reduce the manual cost of running this combination at scale. Rescana takes this approach, as do several platforms with different emphases on the external-signal versus evidence-collection side: BitSight, SecurityScorecard, and UpGuard lean toward outside-in monitoring; OneTrust and ProcessUnity lean toward evidence and workflow; Panorays spans both. For guidance on evaluating these platforms against your specific requirements, see how to compare TPRM platforms.
Frequently asked questions
What is the difference between security ratings and evidence-based vendor risk management?
Security ratings are outside-in scores derived from observable internet data - exposed services, certificate hygiene, DNS configuration, and leaked credentials - assembled without vendor participation. Evidence-based vendor risk management collects and validates vendor-provided artifacts: questionnaire responses, SOC 2 and ISO 27001 reports, penetration test summaries, and contractual controls. Ratings are fast, broad, and continuously updated, but they cannot see internal controls. Evidence-based assessment is more time-intensive and requires vendor cooperation, but it covers data handling, access management, incident response, and contractual obligations that ratings cannot reach. The two methods work best in combination, with the weighting determined by vendor tier and regulatory requirements.
Are security ratings reliable enough to make third-party risk decisions?
Security ratings are a useful input but carry well-documented limitations. Their methodologies are proprietary, so scores for the same vendor can differ across providers and are difficult for vendors to reproduce or dispute independently. They measure observable internet-facing posture, not internal controls governing data handling, access management, or incident response. Most regulatory obligations - including DORA's contractual requirements and the NIST SP 800-161 C-SCRM framework - cannot be satisfied by a score alone. Used as a fast triage and continuous monitoring signal for broad vendor populations, security ratings add real value. High-tier vendors and those handling regulated data warrant evidence-based assessment alongside any rating.
How do security ratings and evidence-based TPRM complement each other?
The most practical approach is to tier vendors and allocate each method accordingly. For critical vendors - those with high data access, operational dependency, or regulated data - use a full evidence-based assessment with a security rating as a continuous monitoring overlay that triggers re-review when the vendor's external posture changes materially. For lower-tier vendors, use the security rating as the primary signal and reserve evidence collection for cases where the rating declines or an incident is detected. This combination maintains a broad, current view of the portfolio while focusing deeper assessment effort where the risk actually warrants it.
Do security ratings satisfy regulatory requirements for third-party risk management?
Not on their own. DORA requires specific contractual provisions - audit rights, exit strategies, incident notification obligations, and documented risk assessments - for ICT third-party providers, which a security score does not satisfy. The European Banking Authority Guidelines on Outsourcing Arrangements (EBA/GL/2019/02) require documented assessments of sub-outsourcing chains and concentration risk. NIST SP 800-161 defines C-SCRM as a formal program with documented controls and a supply chain risk management plan. Security ratings can serve as a continuous monitoring input and a prompt to revisit evidence-based assessments when a vendor's external posture changes, but the documented, auditable assessment process is irreplaceable for compliance purposes.