Every FHIR vendor claims strong conformance. The claim is the easiest thing on the pitch deck; the evidence is often the hardest thing to find. Quantifying the claim before the contract signature is the shortest path to knowing whether the vendor's story survives the audit that will happen once the product is in production.
Naming the evidence sources is the discipline that turns claims into scores. For related reading, the FHIR adoption tracker collects the surrounding material.
What the Claim Actually Says
Vendor conformance claims come in three flavors:
- Base R4 conformance: the product speaks the base spec correctly.
- Profile conformance: the product supports specific profile packages (US Core, Da Vinci, etc.).
- Certified conformance: the product has passed an external certification like ONC.
Each carries a different evidence bar. A pass through the site's FHIR engine scorecard can carry conformance as its own axis with the granularity above.
Ask for the Connectathon Record
HL7 connectathon results are public. Vendors that participate regularly have a track record; vendors that skip usually have a reason. The connectathon scoring is the closest thing FHIR has to a public conformance benchmark.
Ask for the last four connectathon participation records. Vendors that furnish the data are easier to trust than vendors that hedge. For the wider scorecard method, building an honest FHIR product scorecard covers the composition.
Test Against the Public IG Publisher
The HL7 IG Publisher is a public tool that validates a product's response payloads against loaded profile packages. Running a small suite of representative payloads through the IG Publisher against the vendor's demo environment produces evidence in an afternoon.
The exercise sounds heavy and takes less time than the fifth vendor demo would. It is worth doing for the two or three shortlist vendors before the contract conversation.
Read the CapabilityStatement Carefully
Every FHIR server publishes a CapabilityStatement describing what it supports. The statement is not a promise; it is a self-report. Reading it carefully reveals gaps between the vendor's marketing and the vendor's own machine-readable claim.
A CapabilityStatement that omits a resource the vendor claims to support is a red flag. A CapabilityStatement that lists a resource but marks the interactions as read-only when the vendor promised writes is a bigger red flag.
Reference Customer Verification
Reference customers who have hit a specific conformance edge in production are the highest-value evidence. Their stories about the workarounds required, the vendor's response time, and the resolution timeline are the sharpest picture of what conformance really means at that vendor.
Ask specifically for reference customers who have escalated a conformance ticket. Vendors that can point at successful resolutions are usually mature; vendors that cannot point at any tickets are usually hiding something.
Score Against Evidence, Not Slides
Every conformance score should trace to one of the above evidence sources. Vendor slides are not evidence. For the wider RFI-versus-scorecard question that this feeds into, scorecard vs RFI: when to use which covers the sequencing.
The score does not need to be a single number. A range that says the vendor is a six-to-eight on base R4 and a four-to-six on US Core is more honest than a single seven. The range invites the follow-up questions the score should invite.
Publish the Evidence Trail
The scorecard should carry the evidence trail. For each conformance score, a line item names the source: connectathon result, IG Publisher run, CapabilityStatement audit, reference customer transcript. For the six categories that show up in mature scorecards, the six categories every FHIR scorecard needs is the accompanying reference.
Quantifying the claim is the small discipline that keeps the conformance conversation from becoming a marketing echo. Every FHIR platform decision benefits from it.
Sources
- HL7 FHIR core specification of conformance module - HL7 FHIR core specification of conformance module, canonical reference for measuring vendor claims