The 10 essential AI capability questions you must ask suppliers

📨 Subscribe | 🤖 AI Training | 🚀 Courses

 TODAY’S THOUGHTS ☠️

Hey there 👋,

Did you know the number of Workplace L&D Tech and EdTech companies that advertise AI as a core feature has exploded by 255% since 2022?

So we could say every company is an AI company.

But where does this leave us, the humble L&D budget holder, trying to figure out who has the good stuff to help us reshape what we do?

That’s what we’ll explore today with 10 essential questions to ask suppliers about their AI capabilities.

(This is part 2 of my 3-part series on navigating conference season and making smart tech buying decisions.)

Get your tea or beverage of choice ready, 🍵.

We've got lots to discuss!

P.S. Your app might clip this edition due to size, if so, read the full edition in all its glory in your browser.

 IN THIS DROP 📔

• Assessing AI-powered features like a pro

• Finding the real value in your organisation’s AI products

• A prompt experiment to try out in your research

🙋‍♀️ Want to reach over 4,500 L&D pros? Become a Newsletter sponsor in 2025

 THE BIG THOUGHT 👀

Navigating AI In Learning Tech: The Questions You Need To Ask

This is part 2 of my “navigating conference season and making smart tech purchases” series.

I’m aware that’s a terrible title, but my tea just ran out.

I recommend you check out part 1 to build your pre-game approach, aka before you hit the next conference floor.

The goal of the series is to help you:

1. Pick the right partners to explore

2. Know how to assess the AI capabilities of tools and platforms with smart questions

3. Make the best purchase for your organisation, team and goals

We’ll cover number 3 next week with an in-depth guide to choosing the best supplier and platform combo.

Technically, you can use all of this playbook series without attending a conference this year. My mission is to make sure you spend your L&D tech budget on the right stuff, not more stuff.

AI, AI and AI

Yes, you know it, and I know it - every learning platform has AI.

But what does that really mean?

With generative AI evolving so fast, something you purchased last week could be outdated by next month. While it’s hard for suppliers to keep up, it’s good to understand their approach to AI-powered features.

To help you with this, I’m giving you my 10 essential questions to ask suppliers about their AI capabilities.

This list is a combination of curation from this Josh Bersin article, my experience from multiple implementation projects, and too many chats with AI and deep chats with fellow humans.

I like to think it’s been battle-tested!

Btw, you don’t need to ask all 10. Use whichever works best based on your context.

Why these questions matter

I’ve crafted these questions keeping in mind that you’re probably a non-techie.

Depending on the size of your organisation, you can call in help (and def should) from any departments working in tech purchasing, machine learning and AI (of any kind).

But I’m aware that luxury is not available to all.

So, with that in mind, these questions will help you no matter your technical expertise.

Yet, if you want someone to help you with the research, identification and decision-making process with AI features in your platforms, you can book a consulting session with me to deploy my thoughts as and when you need them.

Anyway, shameless self-promo plug over - let’s focus on the questions.

1. What LLMs do you use and why have you selected these models?

Why this matters: Not all large language models (LLMs) are created equal.

Some are great at summarising complex topics, others are better at casual conversation. You need to know why they picked an LLM and how that supports the performance of features across the tool or platform.

You want a reasoned choice based on your industry needs, scalability, data security, and feature performance.

2. How do you select and manage the data for training your AI models?

Why this matters: The quality of data feeds the quality of the AI.

Now, this question is a little more technical and can leave you exposed without the necessary knowledge. So by all means, if you’re not comfortable with your understanding, remove this one.

For 99% of companies, they will use one of the large LLM providers in their stack.

Think of this like a wrapper. On the front-end, you see the brand you’re working with and beneath the surface the tech is powered by OpenAI, Anthropic or Microsoft.

This is standard - you just might not be aware of it.

In most cases, your supplier might have nothing to do with the data used to train the models. If they’ve purchased an off-the-shelf solution from one of the providers mentioned above, this will almost certainly be the case.

That is, unless they’re fine-tuning an instance of the model directly.

Fine-tuning means taking an AI model and giving it additional training in a particular domain. For example, a company may fine-tune one of OpenAI’s models on leadership principles to develop an AI leadership tutor.

This question can be converted to: Do you fine-tune your AI models or use them as delivered?

3. Can you detail the specific generative AI features your product offers?

Why this matters: "AI-powered" could mean anything from smart search to automatic course creation. You need specifics to know what you’re actually buying and whether it solves the right problems.

Ask about the concrete features tied to real user needs (e.g., summarising, content generation, personalised learning paths), and to help your internal team, of course.

Generative AI is only one part of the much larger AI family.

4. How is user feedback incorporated to enhance your AI models?

Why this matters: Again, use this like question 2.

If the supplier is using an out of the box AI model and not fine-tuning, they might have no control over this. But it doesn’t hurt to ask.

Knowing how feedback is used to improve the product is always useful. It doesn’t have to be linked to the AI model itself, it might be how the user interface works to enable you to make the most of those features.

5. Can we talk with customers using your tool and see how well it performs?

Why this matters: Real users don’t have a marketing agenda.

Conversations with existing customers will give you a balanced view. If you do get this opp, ask these people how they’re using features, why these features and the impact they’ve seen outside of “We can create more content faster”.

6. What kind of support and updates can we expect to keep the AI solution current and effective?

Why this matters: With Generative AI, standing still is falling behind.

Without regular updates, models degrade, and tools can quickly lose relevance. We want to avoid this by looking at products as ever-evolving. You want a partner that reflects this.

7. Can you share real-world examples where your tool has positively impacted learning outcomes?

Why this matters: You want proof, not promises.

An AI feature might look impressive, but unless it moves the needle on skills application and performance, it’s just noise.

Here, it’s useful to re-align with the core problem you’re trying to solve with this purchase. Seek to learn where these features directly link to this.

Tip: shut down generic statements like “customers love it” without any evidence.

8. How do your generative AI features stand out from what others are offering?

Why this matters: Everyone claims to be “AI-powered” now.

You’ll get these features anywhere today. So, what makes this supplier different?

What is the unique selling point they bring?

9. To what extent can it be customised to fit our unique needs?

Why this matters: No two learning environments are the same.

A rigid, cookie-cutter solution will create more problems than it solves. This is always an important conversation to have. As I said last week, you want a partner that supports your growth, not another provider who disappears after deployment.

10. What training and support are provided to help integrate and utilise these AI features effectively?

Why this matters: Buying the tech is the easy part.

Getting people to use it properly and consistently is the hard bit. Without structured training and onboarding, adoption will be slow, patchy, or doomed.

Again, this is why you want a true partner.

What’s the approach to work with you to not only deploy this platform, but make it a success?

And no, you don’t care if they have a ‘help centre’ for 24/7 doom-scrolling.

Prompt playground: Try it yourself

Copy and paste this into your AI assistant of choice to experiment.

# Context

I'm a learning and development manager looking to evaluate the latest learning technology. I'm attending a number of conferences over the next few months where I'll be assessing suppliers AI-powered products.

I want to be prepared by understanding the best questions about their product's generative AI features and larger AI capabilities.

# Task

Your task is to suggest no more than 10 questions I should ask as part of a validation process on their products generative AI capabilities.

These should be clear and easy for me to understand as a beginner in the world of generative AI.

Final thoughts

Ok, this is your foundation.

You can and should customise this to your context.

I like to think of this checklist as your BS detector, negotiation tool, and insurance policy all rolled into one. I think in a world full of AI promises, the smartest move you can make is asking better questions.

Next week, we’ll explore how to assess your line-up of potential learning platforms and make the right decision for your context.

→ If you’ve found this helpful, please consider sharing it wherever you hang out online, tag me in and share your thoughts.

Till next time, you stay classy, learning friend!

PS… If you’re enjoying the newsletter, will you take 4 seconds to forward this edition to a friend? It goes a long way in helping me grow the newsletter (and cut through our industry BS with actionable insights).

And one more thing, I’d love your input on how to make the newsletter even more useful for you!

So please leave a comment with:

1. Ideas you’d like covered in future editions

2. Your biggest takeaway from this edition

I read & reply to every single one of them!

 VIDEO THOUGHTS 💾

How To Find Where AI Can Really Help You At Work

One of the most common questions I hear from teams is: Where can AI actually help me at work?

The mistake most people make is jumping to the tool first instead of defining the problem. AI can be a powerful productivity booster, but only if you know which tasks to apply it to.

In this video, I break down a simple decision-making framework to help you identify where AI can save time, improve quality, and let you focus on more meaningful work.

Plus, I’ll share three real-world use cases from my workflow, with specific tools you can try.

🙋‍♀️ Want to reach over 4,500 L&D pros? Become a Newsletter sponsor in 2025

P.S. Wanna build your L&D advantage?

Here’s a few ways I can help:

• Build your confidence and skills with the only AI course designed for L&D pros.

• Become a better L&D partner with the Art of Performance Consulting.

• Get a backstage pass to exclusive industry insights, events and a secret monthly newsletter with a premium membership.