Most HR groups have spent a good period of time constructing a governance framework round AI use. They’ve:
- Drafted acceptable use insurance policies
- Set guidelines about what knowledge staff can feed into giant language fashions (LLMs)
- Thought of the danger of bias in AI-assisted hiring selections
- Explored how they’ll shield the safety of delicate knowledge when it’s uncovered to third-party instruments.
That scepticism is well-placed. However that very same scepticism is basically absent when AI shapes HR’s personal buying selections.
The governance hole no one is speaking about
When an HR chief asks ChatGPT to: ‘suggest the most effective HR and payroll software program for a 700-person UK enterprise’, they get a clear, assured shortlist in seconds.
It feels authoritative. Like analysis. Just like the inexperienced mild to make a assured shopping for determination.
However the identical issues HR leaders elevate concerning the position of AI in folks selections apply immediately right here, too:
- The place did this advice come from?
- What knowledge was the AI device skilled on?
- Is that coaching knowledge present and legitimate?
- Might it’s biased in the direction of sure distributors?
- And who’s accountable if this determination seems to be flawed?
AI suggestions aren’t true analysis – they’re a chance output
AI search instruments generate plausible-sounding solutions primarily based on what they had been skilled on and what they will retrieve. They don’t conduct an goal evaluation of which HR or payroll software program is the most effective match to your organisation.
Analysis by SparkToro in 2025 requested 600 volunteers to run almost 3,000 brand-recommendation prompts by ChatGPT, Claude and Google’s AI. There was lower than a one-in-100 probability of getting the identical listing of really useful manufacturers twice and roughly a one-in-1,000 probability of getting them in the identical order.
However randomness is barely a part of the issue. AI instruments additionally hallucinate – and in a procurement context, that has actual penalties.
Right here’s a situation which may sound acquainted. Ask an AI device concerning the performance of a selected HR system and also you’ll get a listing of confident-sounding solutions. However should you explicitly instruct it to not make any guesses and assumptions, important factual gaps develop into seen. The AI mannequin has fabricated plausible-sounding particulars and introduced it as reality.
Randomness is barely a part of the issue. AI instruments additionally hallucinate – and in a procurement context, that has actual penalties
The content material lure: loud voices, not the most effective match
There’s a second, much less apparent distortion. AI doesn’t all the time floor the best-fit vendor. It surfaces the seller with the most important on-line presence.
Search-integrated AI instruments – like Google’s AI Overviews – pull from what’s out there throughout the online. Distributors who make investments closely in on-line content material and evaluation platforms seem persistently. Distributors who’re genuinely wonderful however much less seen on-line could not seem in any respect.
This creates what is perhaps known as the ‘Google lure’. The abstract card on the high of the search outcomes feels complete, so patrons cease scrolling. They don’t attain the seller web sites, comparability pages, or specialist directories decrease down. And so the shortlist closes earlier than it has totally opened.
The choice making accountability hole
Once we belief AI to decide, who’s held accountable to the end result?
In AI-assisted hiring, HR has rightly fought for explainability. Choices that have an effect on folks’s livelihoods ought to be traceable, auditable and defensible. HR has pushed again laborious on “properly, the algorithm stated so” as a adequate justification.
However “ChatGPT really useful it” is changing into a suitable clarification for a software program determination that can have an effect on each individual in an organisation for the following 5 to seven years.
And the implications of getting it flawed aren’t summary. Distributors are more and more in a position to determine patrons who’ve let AI instruments lead their searches, as a result of these patrons arrive with lists of must-have options that don’t replicate precise enterprise processes.
Whereas AI might need given them a shortlist of methods to demo, it hasn’t supplied a sturdy listing of necessities. And it’s in these conditions – the place the necessities listing doesn’t match with the necessities actuality – that HR software program implementation tasks go flawed.
HR groups ought to deal with AI shortlists the identical means they’d deal with any unaudited output: as a place to begin that requires human verification
Find out how to confidently use AI within the shopping for course of
Used properly, AI instruments velocity up the HR software program shortlisting course of. They do a fantastic job of producing analysis questions, transcribing demos and drafting comparability matrices.
The device isn’t the issue: the problem is the absence of the identical vital framework that HR applies all over the place else.
HR groups ought to deal with AI shortlists the identical means they’d deal with any unaudited output: as a place to begin that requires human verification, not a conclusion.
Refine your prompts by working in components out of your necessities listing, and explicitly instruct the AI engine to think about HR methods’ capability to fulfill these necessities. Analyse the responses towards your listing of must-haves. Doc who made the ultimate determination and on what foundation.
HR people might want to do the foundational work that AI can’t do: defining necessities earlier than you open a search device, involving the appropriate stakeholders and operating a structured analysis the place distributors are assessed on the identical standards, in the identical means.
The time saved on the analysis stage is rapidly misplaced in a failed implementation.
Actionable insights
- Make sure you’re making use of the identical rigorous AI governance you apply to folks selections when buying software program
- Deal with AI outputs as a beginning speculation, defining necessities earlier than looking out
- Guarantee HR people do the foundational work that AI can’t do
- Hold closing selections with individuals who could be held accountable for them.
Ciphr is a UK HR and payroll software program firm that helps mid-sized organisations join the methods, knowledge and other people behind nice work. Be taught extra at ciphr.com.
