AI Agents in Action: Shaping the Future of Human-Machine Collaboration

As AI agents shift from buzzword to business imperative, organizations are asking not just if they should implement them—but how. During Innovation Summit 2025, experts from banking, data strategy, and enterprise tech pulled back the curtain on where companies really are in their AI journeys, what makes agents effective, and what foundational work can’t be skipped. Whether you’re piloting internal tools or planning customer-facing automation, the insights shared here offer a grounded, strategic roadmap for getting started—and getting it right.

Key Summary:

1. Don’t Skip the Foundation: Data Quality and Governance Are Non-Negotiable
   AI agents are only as good as the data they rely on. The panel stressed that success starts long before implementation—with ensuring your data is clean, governed, and contextually complete. This is especially critical when integrating multiple sources like PDFs, audio, or system logs.
   
2. Start Internally, Not Externally
   High-risk, customer-facing deployments are tempting—but premature. Many organizations are finding success by first using agents for internal training, service desk support, or HR tasks. These lower-risk applications build organizational confidence and refine processes before scaling outward.
   
3. Small, Purpose-Built Agents Outperform General Solutions
   Rather than relying on massive, general-purpose models, the panel advocated for building compound agents tailored to specific tasks and datasets. This modular approach increases speed, reduces hallucination risk, and allows for precise tuning as needs evolve.
   

Change Management Is Just as Important as Code
Building an AI agent is the easy part—getting it adopted is the real challenge. From executive buy-in to employee trust, organizations must address the human side of transformation. Framing AI as a tool that enhances (not replaces) roles is key to widespread engagement.

Thank you very much for joining this session this is all about AI agents uh got two two great panelists here to have a have a conversation but obviously that conversation can extend to all of you so we’ve got a a mic where we can run it around if other people want to have a have uh ask questions but before we get started let’s just so we understand where we’ve all got to who here has actually built an agent okay a few of you all right great so few people have got some experience but a lot of people are here clearly to learn what works what doesn’t work in what is very clearly an emerging marketplace and no one has all the right answers there’s no such thing as best practice there’s only where we’ve got to at the moment and it’s and we’re still evolving that so we’re all here to learn collectively about what we can do better to to make agents work so let’s first of all let’s meet the panel so Heather can you just explain just a minute or so about about your background 

absolutely can you guys hear me working all right my name is Heather Maples i’m with Dollar Bank uh my background is a little bit interesting so I didn’t start out in technology i was actually a sales manager about 13 14 years ago running a fintech company and learned Salesforce there i became that accidental admin so with that I really decided to take my career and change from sales management into the Salesforce ecosystem and started working at Suma Technologies so friends with the Hike 2 team here pretty closely and just found a passion for the platform and did all of the trail heads taught myself how to code do all the amazing things that Salesforce offers and then went to go work for PNC Bank did a green field implementation there and found an analytics gap and introduced CRM to the bank um when I left there we were across the entire nation in every branch leveraging analytics to be able to help our bankers do their jobs more efficiently um recently joined Dollar Bank as a Salesforce platform owner um another Greenfield implementation that I’m really excited to bring all the knowledge forward um it’s a pretty cool journey one I never thought 12 years ago if you asked me where I would be owning a Salesforce platform for a bank is is not the answer I thought I would give um but really excited to be here thanks Ian 

so Andy just a summary about a little bit of your background just to set the context 

so Andy Stahl I’m with Data Bricks uh I’ve been in analytics more years than I care to think about uh if you’re a Pittsburgger you probably will appreciate this my favorite dinosaur is Dippy okay uh been building and deploying analytics for a long time was part of a 23 person machine learning startup that was acquired by Salesforce mr coulson there knows and I’ve been part of that uh and that was the very first Einstein offering back in 2016 and John and I and some others helped scale that business to an $800 million business within Salesforce uh now my job is a combination of public speaking in teaching within data bricks customers as well as partners as well as internal and I actually do have some client-f facing responsibilities and I implement agents i implement AI data intelligence uh so hands-on doing work my coding sucks but thank god we have agents that could do that for me 

excellent thank you um I guess two seconds on me three numbers 10 20 30 uh 10 years as an Accenture consultant i’ve written 10 books on business change and compliance 20 years as an Accenture so 20 years as a Salesforce customer so I’m allowed to drink now and 30 years on my soap box saying why business analysis is important and interesting interestingly enough business analysis suddenly is becoming even more important when we’re building agents so internally we’ve built eight eight agents they’re all live got another eight in in production so we are we’ve been on this journey for a short while and we’ve got some experience about what works and what doesn’t work but before we launch into some of the things about what does and doesn’t work let’s Heather can just tell me where you are on your journey from no idea what to do thinking about it deploy uh building and deploying where where are you on that journey 

so we are greenfield implementation right now so I’m in the don’t know what to do moving into the thinking about it range and when we start to think about how we would leverage it I’m finding those use cases internally that will help us do better internally um obviously as a highly regulated industry there’s some extra things that we need to look at and think about but yeah we are in that ideation phase um I have my eyes definitely on it so someone might go well that’s great but why on earth are you on the panel if you’ve never had done it with agents which I think you might that’s a question for the audience but I think you’re at the the really important part where many people are which is going okay we can’t just throw these things into the business we’ve got to think about it from I know a compliance perspective so there I think your insights into what it takes to actually get the buyin from senior management I think will be really important i think it’s that as well as being that green field implementation and we’re working on a data migration strategy right now to ensure that our data is clean and ready so when we do bring the agents in we are hitting it headon versus having to do a bunch of cleanup and data clean cleansing prior to it we’re going to be able to jump straight in okay so getting some of the foundations in place and we’ll pick that up a little bit later andy what I mean you clearly you you’ve been talking about agents and AI for a number of years but I where are you in in terms of data bricks internally but then think about some of the clients you’re talking about where do you think they are in terms of that journey most of the clients are relatively early and here’s the problem if you look at maturity curve and I’ve started talking to them about maturity assessments they they’re still half of them are still out of the gate they’re not they haven’t done anything they they’ll bring in a chat GPT or Microsoft co-pilot or something and you know three people play with it uh that’s great but that’s not how agents or how the technology is going to transform and I specifically use the word transform uh you think about I tend to think in terms of the agent or the use the term AI as your new UI so I think in terms of particularly I’m old I’m cranky I have a dinosaur and but how do I get me to use it okay I can get my 27y old to use it right but getting me to use it and there’s a if you look at the bell curve of employees by the way the older the employee is the more tenure the less successful AI is with them by the way statistical studies have shown so how do you get it to have value for me how do you have it value across the organization so I think in terms of that and I start for that as the result because I can build anything building it is not hard building an agent a day deploying it getting it used really hard so if I intend to be successful I go from the the other side how do I get people to use and consume and that’s much more understanding the psychology of that and then from there work backwards of what it needs to be and incorporating it into a way that it’s part of their job the the model I use in thinking about things is Amazon’s recommendation engine which is others like you it’s widely used uh if I tell my wife that’s a recommendation engine it’s going to give her a guided insight on what to buy she tells me to go f off go back to my reading but people use millions of people use that every day and they don’t think in terms of that it’s something that’s useful and valuable so you have to think in terms of putting it in like Amazon has in their page as part of the UI to make that happen in consumable that will change how you think of applications so let’s pick up there are a number of points you let’s pick up some of the points there i think Amazon and you kind of understand what the recommendation engine’s doing but I think with agents there’s a there’s a level of confidence we’re not getting there which I think senior executives are going okay I’m not confident that this is going to come up with the same answers consistently so I think we’ve got a slightly different challenge there as well which we need to get over that’s an engineering problem what is it okay okay so if you think about what you said consistency of response okay there’s a couple of things associated with that you’re never going to get 100% let’s be real okay and it is remember it is prediction okay so it’s the So if you think about what we’re doing part of the thing I would challenge is I would not look at the mass market LLMs who cares if I’m building an app for in my internal to my company if I’m not into baseball who cares who won the 1974 World Series chat GPT can give me that who needs that so I’m going to construct my agent and and I and by the way it’s not one agent it’s a series it’s compound it’s multiple agents doing very specific things tool and tasking purposely built personally I’m a fan of small language models that are much smaller i’m not sure it’s the large language models we should be concerned about surely it’s actually how we’re instructing them correct and that’s it and if it you can the and it’s the training data you put into there then what you have to do depending on that data depending on the complexity partic and remember when you think of data I’m as guilty as this is everybody we all think in terms of the classic Salesforce ERP workday SAP tabular data right I call tabular rows columns goes into a spreadsheet into a data warehouse that’s a tiny part of the data associated with it so you have to think in terms of PDFs JSON audio call logs all this stuff that requires a different thought and how you provide data hygiene particularly if you’re looking like contracts contracts big issue you want if I’m going to do a contract recall I have to have the exact contract right contracts not always one document it’s multiple documents so you got to start thinking in terms of an ontology that links all the documents together so you can pull them back and have a summary of everything in that contract otherwise your legal do your legal people or are going to have to go back and do it manually then why bother so if you do all of that then on top of that you need to have the ability to trace and have an audit trail because every time I get a bad answer think of in terms of reinforcement learning it’s a bad answer somebody tells me it’s a bad answer how do I fix it if I don’t know what document it went to I don’t know where it is i can’t do that so if I engineer that reinforcement learning and I have the ability to trace I can track the document so what you’re really getting into is governance governance not just on the tabular data but across all your data so it requires a much more engineering software focus to be able to do that you can substantially reduce your risk of hallucinations to be able to do that but it requires planning and understanding if you don’t you’ll spend a lot of money and you’ll get nothing okay well I agree with that i I wasn’t agreeing with you at the beginning but you’ve all you’ve argued me around to actually yeah it is an engineering problem yeah

so Heather with I guess with that as a backdrop one of the things we talked about was improving data but what are the use cases that you’re currently thinking about let’s see where say where you’re starting so everything that he just said is why we are taking it slow and I didn’t understand half of it but this is why we hire the people that do so my initial use cases are very basic training and this is something I think that we can really bring on when we implement Salesforce is to have those agents available to help our training go faster we just heard from Shannon i caught the end of her session and she was it was awesome to hear about adults and learning and how we are going to need that top down buying by applying the agent into our training modules and giving them access to this so they can train as they’re going and it helps them do those repetitive things i think it’s a great starting point to introduce it to start to teach the model and then to get that adoption and that buying internally and then second to that as a one that’s high on my list is from a service perspective internally instead of using knowledge articles using those agents to create the case and then bring forward the information and help our service teams solve cases faster and hopefully bring down the transfer of calls and make it that first call resolution so that’s interesting a couple of those well both those use cases are both internally focused yes i think I I think we’re seeing a lot of people looking at agency and going “Well we can put them on our externally facing website and they can answer everything.” That’s got to be the highest risk area in terms of both liability in terms of hallucination and also you’re not going to get feedback from customers they’re just not going to talk to you but if you’re going going to employees yeah lower risk smaller subsets you can build tighter agents and I think that’s the mistake we’re seeing a lot which is people hitting for the taking your baseball hitting for the I don’t know what the term is now hit hitting for the home run rather than actually going no I’ll just take a bunt here we’ll just get to the next we’ll just get get to the first base and I think there’s quite a lot of people not thinking about the strategy correctly I think your approach is absolutely right let’s think about employees let’s think about where we can actually help and engage them and take some baby steps so is that what you’re seeing with with the customers you’re talking to partially so customers that I’ve seen are trying everything it’s It’s no differently than the machine learning craze of 2015 2016 people trying to do different things so they fail it’s the It’s watching your children grow step stumble fail you try not to hold them up you let them fall and then they learn to get up on their own okay can I start for a minute what do you mean by fail they try a project like chatbot okay we’re going to throw a customer service chatbot out there okay i’ve not met a chatbot yet that I like okay you know I got a uh American Airlines do you want to rebook through the chatbot yeah I’m going to risk my flight on a chatbot okay it may make them happy it may work but it’s stupid so it’s understanding that human condition to be able to do that so they all try something because it’s need innovative and somebody get you know has an objective to do it but it comes back then what happens is corporate risk gets into involved like what does that mean to our customer experience okay am I going to drive you away to go to United not there any better by the way and so you got to start thinking in terms of bringing it back so internally we decided we’re going to build a chat excuse me an agent for sales okay we looked at agent force we looked at a variety of technologies and we’re geeks so we decided to build our own okay it took three times as long to build it okay and the biggest problem was what’s really important and what we found was the vast majority of data that we needed for the chatbot didn’t live in Salesforce it’s supplementing what a salesperson needs so for instance compensation data none of that lives in Salesforce in our case it lives in Exactly all data about customers and consumption and utilization that lives in a data bricks instance so but we wanted to put it how would you use and consume it so thinking through that UI to put it there so we spent a significant amount of time understanding and using it and then when we rolled that out to the pilot group we we tracked feedback okay and every time somebody clicked the down arrow you got a slack within a minute of somebody a human okay saying “What the hell why is this wrong what’s your concern about that?” And we addressed every single one of those to understand and to define and is it a data quality issue and by the way the bulk of it was data quality and so to give an example I was looking at telco customers and I wanted to uh understand how other telco companies are using product versus ours the one I was looking at and every answer came back was AT&T now AT&T is a massive customer let’s not hide that but after the first 15 answers were AT&T it’s worthless to me i need to know what maybe Rogers in Canada or NT in Japan or whoever right telefonica and so we realized that if we because it was overly weighted based on the volume and such we had to train the bot to be specific about more detailed on that to provide a quality answer back and it just took time okay so I think this is getting us nicely into what we think the foundations are for for trying to build these agents so you clearly talked about data quality um what are the other things Heather what are the other things that you’re considering as as as we you’re leaning into agents one of the big ones is the security around it and ensuring that it’s going to send us back those appropriate responses the last thing you do want to do is call your bank it’s your bank right it’s your money and you’re going to say something and they’re going to be like “Oh let me just look that up on my little chat ball thing.” And it sends back some whack answer and now you’re giving the client terrible information so ensuring that we’re teaching it the right thing so that we can have those educated conversations and we’re locking that security down so it’s not leaking okay so there were two things there one I think is reliability and confidence correct which I think Andy talked about earlier in terms of trying to getting that feedback and again I think that’s really important the idea of if the results aren’t good let’s very quickly understand they’re not good so we can fix them rather than uh just uh the person just stops using it and you don’t realize okay so that’s the first thing security was the other issue which was about making sure that you know customer data doesn’t leak correct making that’s number one obviously making sure that PII is locked down like the Vatican we don’t want anybody being able to see that stuff um it’s I always go back to data like data data data it has to be good and it has to be clean and it has to be accurate before any of this is going i’d add one thing it has to be governed yes okay so it’s it if you think about it it’s a combination of hygiene management and governance that will ensure you the highest success in doing it it’s and from there it’s just operationalizing it which is don’t get me wrong operationalizing it at scale can be challenging but there’s tooling and things that allow you to do that the key is if you don’t get your data right the foundational pieces and you got to remember data is not generally on one specific platform i might have it across other places so the governance becomes an issue so then you have to think about do I centralize my governance in my data in a specific area and again the skills for hygiene what I do in my tabular is very different than I do good example i was talking to a client about the contract use case they wanted to to do summarization he has 40,000 legal documents right and he first said he says to me “What did you think about the use case?” They said “Great use case won’t work.” And he looks me and says,”Wh by the way I found out afterwards they spent eight months with a partner trying to get it to work.” And he says to me ‘Wh i said ‘Well you have contracts first contracts date from the 80s last contract dates from last week some are docuign most are PDFs that are scanned okay that somebody typed on and then somebody addended put a handwritten note on it and everything else and now you’re asking an AI to read somebody’s handwriting and oh by the way that contract has 14 different addendums from different years so it goes to the ontology problem said what you need to do is you need to spend time on your OCR and then you need to build the ontology and then I can get it to work but so the point there is data is not just data sitting in a database somewhere it could be could be voice absolutely could be could be text it could be unstructured PDFs then the other thing which we probably haven’t picked up on this those documents are probably not written with an AI in mind absolutely not they’re written for for a human yeah by the way the other part that he wanted to do is he wanted to take what was in the contract and compare that to a video picture of a device taken by a drone so if you think about what he’s really talking about in this scenario is a bill of materials as designed as built as maintained comparing to the contract because that leads to revenue leakage if the contract says this and you have different things on the device he can go back and charge revenue for so it’s a a hugely important use case for him but it’s bound the issue for him resolving it is a combination of data quality both on the pictures because you got to be able to be accurate in discerning what that device is and then comparing it to the contract document and being accurate what’s in the contract all right that that sounds like a great use case i don’t think it’s use case number one though it was it still is okay all right that’s one of those you just back away gently that’s fine that’s not Well we’ll I’ll build it for him makes me think I’m thinking okay but so but let’s picking some of the points out of that one is data quality is not just there are different forms of data that we need to worry about the second is we need to make sure if it’s if it’s procedural or do it’s PDFs it has to be written with an AI in mind which is like a 14-year-old who just reads things everything literally when I I was saying to my my son Max said “What are you eating chocolate?” He said “It’s a Wednesday.” Well what what’s that got to he said “Well you said on Tuesday not eat chocolate.” No that’s not what I that’s not what I meant i meant on Tuesday don’t eat chocolate not on Tuesday don’t eat chocolate but that’s that that one comma makes a difference so we again we need to think about some of that documentation and then as you said put a change cycle around if we are going to improve it how are we going to go and improve it again probably huge issue at a bank where you think how much documentation’s out there oh for sure yeah as soon as he was talking about contracts I was like no we’re not touching that one yet not yet but but but you will get to it yeah indeed so when you start or when I start to think about implementing these agents it’s it’s this stuff like okay I could do it for next best product i could say who has what product and what would where do they fall and what should they we should pitch to them next but the contract thing and when you start talking about the really sticky stuff that’s where I I would love to talk to other banks and see how they’re implementing it because we would be like “Nope.” You know the Homer Simpson thing where he backs into the bushes slowly that would be our approach on that one but it’s a risk decision it is yeah yeah and we’re not very riskaverse but but but you’re riskaverse because you’re because you’re a bank right um but that doesn’t mean that there aren’t good use internal use cases that you could use them for so I think saying risk is risk shouldn’t be a reason for saying no it should be a reason for saying not yet or what are the early what are the early steps because you have to make you have to get started absolutely i mean a the agents are future it’s going to happen they’re sticking around they’re not going anywhere so it’s absolutely starting at zero and just easing our way into it teaching it from the inside it’d be amazing for us to have a bot that’s externally facing so that our clients could do more self-s served but not yet then we got a question sorry yeah could you run a mic thank you here just sorry so just so we make sure we have on the recording

thank you um uh Andy had started kind of talking about this with like setting realistic expectations right this number one use case is oh I want to compare these different types of data and also you know all the data is in different formats and also the data quality is pretty bad right but that’s the number one use case so I guess one of the questions I have is like you know from an internal perspective from a consultant perspective whatever perspective how do you set appropriate expectations with what it actually might look like to try to start the process of implementing AI because realistically you want to start small and work your way up but that’s not really what clients want more often than not well it depends okay the if you’re thinking in terms of that that’s not a practitioner discussion that’s an executive discussion and really tied to strategy okay so the discussion I had on the contracts the person I had the discussion with were two people the CFO and the CIO okay not a practitioner or anybody else and the reason that was their number one use case is that can contribute significant amount of revenue to their bottom line so they’re looking at it from a value impact realization that’s a big one much more so than virtually they’ll play with co-pilots and kick the tires and have people write some letters and but that’s meaningless on a value realization perspective not unimportant because you’re trying to build muscle memory you’re trying to build some skills but if you’re looking at being transformative right and in their case they’re thinking very transformative how is this going to change my business right and it took a while to weedle that out of them because they it’s not words that natural to them okay so think in terms if I’m having a candid conversation with a business leader a seale executive whatever what are you trying to do with the business what’s important and what is your threshold for risk back to that point this company I mean you we say contracts it’s risky absolutely if it if you’re thinking in terms of a bank and I’m a dollar bank customer okay you think in terms of customer experience the last thing I want is somebody telling me that my money’s missing it may be there because it’s the wrong result i’m probably you know I’ve been a dollar bank customer for over 30 years they don’t want me to pull my money out okay so that’s a risk that you have to say from a customer experience perspective so think in term you know generally if you look on risk there’s only three reasons to do agents let’s let’s cut to the chase agent reason number one does it make me money my use case on contracts it makes money for them okay they think reason number two does it save me money and usually save me money is getting rid of people let’s not let’s not sugarcoat that and reason number three does it reduce my risk rarely do agents reduce risk let’s be let’s call that one out but you have to look at that that value of the first two versus the cost of risk if you have that conversation with a business leader right you’ll be able to determine what their threshold is and where they’re willing to invest in that from a consulting perspective otherwise you’re going to have a lot of good conversations and lunch and nothing’s ever going to get done okay but but also that is it’s also painting what what the where you could get to from a vision perspective and then what the steps are that you need to get there and I think one of the phrases you used earlier was build that muscle memory so let let’s not try and build the most complex one until we’ve really understood how to how to build a thinking for the end goal in sight and work backwards correct yeah and take some baby steps there um and some of that maybe so as Heather you were saying is actually getting some of your senior management on board in terms of what are the things what what’s the the work you have to do before you even get started so we we talked about it um about data quality and clearly big issue um I would also say understanding business process because an agent has to go and follow a business process ideally and therefore if they are not well understood I’ll challenge you on that i’m happy to have that challenge yes okay i’d argue understanding the business process yes I may change the business process specifically because of agents are re-engineering and that’s transformative and that’s much more No I get that but but you’ve got to change it from to something so the definition of but we need to even if it’s transformed we still need to get to a point where we know what that agent is expected to do um there’s quite a lot of agents know your business they can just work it out [ __ ] i’m sorry that doesn’t count that’s not that isn’t true um you think of if you think of an agent as digital labor you wouldn’t get an intern into your business and go “Fantastic there’s the phone you know what our clients are like knock yourselves out.” You’d actually say “Okay these are the things I’d like you to go and do.” You’d actually give them some instructions um at in the 1990s for some of those who were around that time um there were loads and loads of procedural documents there were no process diagrams there were no computers i remember my first job was inserting new pages into procedural documents for ISO 9,000 okay um eventually we went pro those big procedure documents no one reads we ought to have process diagrams so people could understand things it feels like we’re falling into the same trap all over again by going the way to build an agent is you just write loads of instructions rather than going let’s there we go so let’s think about what the agent’s meant to be doing now for a bank I’m hoping you’ve got lots of your your processes well documented you’re regulated but for many organizations they haven’t and that therefore that’s another of those foundational activities you can do even before you start going with agents so we agreed i’ll agree on that good okay so and then the then the last thing I think is if you’re expecting agents to reuse some of parts of your application they need to be well documented and often we’re not seeing that so again if you’re expecting to reuse workflows or reuse uh different data structures if if you haven’t got that well documented that that’s going to be another blocker so where where are you on that journey in terms of you talked about data but where are Heather where are you on that journey around process and uh systems we’re in a unique spot now that we’re able to we’re redefining process because we’re implementing Salesforce so with that we’re going to be able to naturally design the new processes document those well I really think we’re at a pretty cool advantage like completely green field we get like the world is our oyster and how we want to design the entire platform and that customer experience and leveraging tools like the agents to enhance it so are you looking at redesigning the business with agents in mind it is in the me yes I can’t speak for the bank so because it’s green field and dollar’s been around for 170 years 75 years they’re stuck in their ways like most financial institutions so it is a part of the process and the learning process of them adopting change and so these new processes that we’re implementing with Salesforce in the back of my mind I am keeping in mind that we will be implementing the agents in the future so I’m preparing it now so should should and or when we go down that road we’re prepared yeah okay that’s so there’s there’s there’s not just techn it’s not just technology or actually it’s about engaging some of the senior management and getting some education there correct okay so I I’m going to toss a statement out the cost of inference the cost of doing this building the the intelligent answer is going to zero open source is going to crush the models and drive the cost of being able to do decisioning to zero the cost in doing this is not going to be anywhere close to that the cost is in your data quality okay it’s in your processes it’s it’s in your governance it requires discipline it will work better in regulated industries because they have to have you do that what I have found a couple of rules that when I work with data when I work with HR data using workday as a proxy it works better because generally the data coming out of there isn’t bad my policies and procedures are defined I have things that can work really well to so to build internal chatbot around HR and policies doable it’s a great way to learn muscle memory moderate value particularly if you’re thinking of employee experience Right but it it can work and it’s relatively low risk financials my SAP data my CFO signs a statement every month every quarter and if he if it’s wrong he goes to jail that data quality is moderately good my financial processes generally are pretty good i have documentation i have auditors that track all that again another area CRM welcome to the wild west okay and the further you go from having clearly defined documented processes with quality data that’s when contract you would think contracts is is good quality data they are if the data quality if the document is good okay so having that is really interesting and if you’re thoughtful it can really do be very impactful and efficient if not I probably shouldn’t say this it’s the consultants will make a lot of money by doing it because they’ll clean up your data for you and they’ll document your processes for you and remember it’s not the traditional tabular data warehouse stuff it’s the it’s that PDFs and JSON and all that other stuff and that’s where people don’t have necessarily the skills internally in many cases to do that because they threw it up on SharePoint for the last 30 years so but I also think that the speed of Yeah yep the speed that that things are changing is actually also driving a different dynamic as well because you you listed off some things which is I know PDFs and data but I’m going to add video into that and audio and audio into that relative relatively quickly so the question at the back there yeah Andy you talked about this before you know let’s take the example of contracts um so most modern organizations use contract management systems have you seen people actually um getting really good at the onlogical work of connecting what is in a contract system to the unstructured data in a way that actually makes that useful uh I’m going to give a vendor shout out and it’s not it’s not us by the way okay it’s not data bricks so I have thought about working with Neo4j it’s a graph database so I think you’re going to have to look at your tooling to do that i don’t not I’m not really deep conversant in like an isertus or some of the contract management systems but I think you’re going to have to if you’re going to get into that and you’re going to look at it you’re going to have to go to something like a graph rag type of approach and there’s a variety of different toolings out there i happen to know some folks at Neo4j that I’ve talked to like hey we haven’t talked in two years or three years tell me what you’re doing and explain to me why this is interesting because I could potentially see value in it so I’m sorry I can’t be more helpful than that but I think you’re gonna have to to bring in that level of expertise specifically on that to drive that so I want to I want to change track and thinking about careers in the future but any other questions from the floor before we launch into sort of a different direction

uh yes so I was thinking you know we’re talking a lot about some of the things that we need to do to build these agents from the ground up i’m kind of coming into AI at a base level are some of the out ofthebox agents that might be sold to my company um do they come with some of those challenges already sorted out or is it from the ground up no matter what agent you buy i’ll answer that so if you think about agents there’s really three approaches to agents today there’s OTS offtheshelf there are technology vendors some big platforms some others that are building agents they have relatively different and every one of them has a different set of limitations uh some of them follow I’ll call what I call the palunteer model palunteer will has really great technology and it’ll deploy a bunch of engineers to build it for you okay so be careful when you say it’s offtheshelf because a lot of it’s some of that uh then you’re starting to see the growth of what I call marketplaces so there’s think of it as like the Salesforce app exchange where you can download there’s a bunch of stuff out there on that and then there’s the other side and I I like to build let’s not so remember my bias i think particularly you’re going to find if you’re going to be particularly at scale you’re gonna have to build and again I believe in a what I call a compound approach so if you think of building an application there’s tools there’s personalities there’s all sorts of things and I use very different specific models to engineer so I’ll give an example we have a bot internally that we actually it’s a product that we built for customers it’s called Genie it’s a horrible name we’ve copied I don’t know if we copied the name from Salesforce but we have Genie okay Genie is at its heart and soul text to SQL now we’re adding some deep logic deep reasoning and things in there so you say it’s one bot that does text to SQL no it’s not it’s eight different LLM models as a function of that engineered so you know you look at it you see a user interface it’s one but it’s very specific to that and what we have found that when you do that from an approach perspective though the engineering effort is more significant one it runs more efficiently than a single so if I compare that to chat GPT or something it runs faster it costs less okay and it’s higher accuracy so the building of the multiple con use the term compound putting it together will get you there but again it goes to my point of being thoughtful because you’re now constructing and you’re doing that it also reduces your risk if particularly on an internal application because you can tune it and as you develop my using my term muscle memory it allows you to start thinking of how I’m going to transition that to my others like you bought this product approach for Amazon so let me take this from adding on that but taking from a different perspective which is yeah there there are certainly AI small limited AI solutions and if it’s in the productivity area which might be emails and so on but if you’re thinking about the core line of business type agents it’s unlikely that you’re going to find all the ones that you need that that support your particular industry or the particular issues so eventually you’re going to run out of those and go well actually we maybe we need to go and build them so let’s start building them and understanding them how how they work in the first place so therefore not building from the ground up in terms of choosing a large language model but maybe starting at the agent framework level where you can start to build agents relatively easily and again not a pitch for Salesforce but that Salesforce and Microsoft Copilot their approach of here’s a platform and therefore you can actually you’ve got the components to go and build an agent is a good way of getting started and start to start to take some simple like line of business type agents and then you can grow those so I think longer term we’re going to see that that there will be you know agents that we’re building and if you think about the evolution of SAP or Salesforce it went from we all used to build all the applications then then then there were large applications available i think in the future we’ll end up with all of our core applications have got most of the agents we need embedded but we are multiple years away from that so I think at the moment we’re at the stage of look at a platform don’t try and build it from the ground up but look at a platform that enables you to construct the agents that you want i’ll add one thing i talked I spend I spend very little time in Pittsburgh though I live here uh I spend a lot more time talking to my friends out on the West Coast from talking to my buddies at Google and talking to my buddies at Snowflake who by the way are my direct competitors okay I can share with you we’re all thinking that this stuff is way too hard okay let’s so let’s be very clear so what you’re going to see is a collapsing of the culture around agents to build directly so think I’ll use the term AI database so if you think in terms of the data structures where because that’s where the data is going to be held and remember that doesn’t mean it’s all tabular it’s again it’s more of a lake lakehouse environment where you’re putting everything in it makes it much easier to govern and you’re going to build the agent functionality there you’ll have all the ML ops to be able to support and scale that built into that structure and then what you’ll be able to do is will be think in terms of a no code low code kind of environment so you’ll be able to say I want this and I want that and be able to do that and that will probably get you to 80 to 90% of what will be done as long as that data is near that platform or able to federate to that platform and be governed by that platform so for Heather that means I guess agent force is your your your on the core Salesforce platform so it may be not the not the only answer but to start with the easy steps is agent force yeah definitely that’s the one I would that’s where you’re going to start that’s where you’re going to start but remember data has gravity and again I think agent force is great for a limited range of applications the that’s what we said it’s about where you get started absolutely where where we end up we could be somewhere else so just thinking about just future proofing your career so like Heather you’ve you’ve had an interesting career trajectory which started as you said accidental admin which all the way through now to platform owner what are the sorts of skills that you think you need to have in place as we move into this new world wow i don’t think I can keep up with all the different skills that are going to be needed i need to learn more about what Andy’s saying and understand i’m not I’m not sure you necessarily do but we’ll have another conversation about You’ll have a very different You’ll have a different opinion when you hear from me on that go ahead okay feel like I’m in a tennis so when I advise kids when I advise kids in college Andy just let let Heather finish oh sorry so um I forgot where I was going with it you were saying about the sorts of skills that you think you you need to Yeah so memory should be one um so from a developer standpoint I think that people are already using agents to help define code i think we’re always going to have to have that human element inside of it but I think it’s going to help us execute faster um but again teaching it and ensuring that it’s building a flow the correct way so we’re not implementing something really wonky and it’s going to do something i think it’s just defining our skills and learning how how to teach the models so versus me learning exactly how to code a flow it’s teaching myself okay how do I need to work with AI to be able to create that flow so using AI using AI as a tool using it as a tool for sure but um I think that human element combined with the technology okay but but but also let’s think about some of the other things you’re doing you’re you’re having to educate senior management you’re having to do a lot more business analysis and data data analysis so the some of those higher level skills rather than developer skills as well for sure yeah it’s it’s constantly changing i don’t know about you guys it’s really hard to keep up with okay now I need to go down this path and I need to learn this or I need to go down this path and learn this but it’s definitely um constantly educating yourself where the industry is heading the specific industry you’re in i look at other highly regulated industries to see how they’re managing it and so it’s yeah it’s just a combination of just keep running just keep running i’ll give you two that not natural to most people tech skills programming etc table stakes the two I advise people kids to particularly in college today first one financial accounting world runs off of accountants I’m sorry and the key in it’s not just understanding the accounting it’s understanding of risk because if I’m going to be developing and deploying systems I got to understand what the concept of risk is and how to manage that particularly if I’m going to be in an executive level I’m more managing risk than anything else second thing cognitive psychology because it’s concept of getting people to use and consume and work as part of the process i can build anything tech skills i can I can sit down with a bunch of folks we can engineer it right and if no one uses it what is the value of it other than I may have gotten paid okay i don’t get my second contract i don’t drive my business value it’s the consumption side so remember I started my conversation i said my favorite dinosaur is Dippy the reason for that is this as an 18-year-old I was in school i was in cognitive psychology they made me go down to the Carnegi Institute to look at the dinosaurs as part of a cognitive psychology class and they learned the history of Dippy and Andrew CargI and all that happy stuff and they came back and said to me I we I was curious i’m like why would you make me do that i’m 18 years old i don’t give a [ __ ] about dinosaurs and what they said to me is human creativity peaks at around age five age four you love dinosaurs age five you get to kindergarten and they tell you cars can’t drive underwater and rockets don’t go through land why because physics because of gravity and all this other stuff right so if I want to be creative and understanding and want to be able to to to have be impactful I want to think like an five-year-old i want to have a favorite dinosaur and I’m going to act like that and it allows me to reinvision the business process and that’s cognitive psychology so that that feels like a perfect place to finish which is it’s actually about business skills and people skills rather than technical skills so again thank you so much everybody for for joining us for this session