Welcome to the Data Cloud Podcast. The next episode features an interview with Liam Hynes, Global Head of New Product Development at S&P Global Market Intelligence, hosted by Dana Gardner, Principal Analyst at Interarbor Solutions. They discuss how S&P Global Market Intelligence uses AI to analyze corporate earnings calls to guide and improve financial reporting.
These insights help businesses enhance communication, refine executive performance, and predict market outcomes. The conversation also highlights the use of Snowflake's Cortex AI platform, and the importance of data-driven decision making in the financial sector. So please enjoy this interview between Liam Hines and your host, Dana Gardner.
Dana Gardner: Welcome to the Data Cloud podcast, Liam. We're delighted to have you with us.
Liam Hynes: Thank you, Dana. It’s a pleasure. Looking forward to it.
Dana Gardner: It's a very interesting use case. You know, among the most promising ways that data science and business intersect is when an entirely new service or use cases can emerge.
And you've been able at S&P Global Market Intelligence to identify an underutilized data resource of the legacy transcripts of corporate results reports to financial analysts and create new insights for business leaders. These new services show innovative ways that AI and human behavior can reinforce each other for multiple benefits.
Tell us how you're using technology to improve how people communicate.
Liam Hynes: Well, that's a great question, Dana. So how do people better communicate? The genesis of all this started around 25 years ago when we were analyzing the Enron case study. Analysts on the earnings call with Enron were asking the executives questions.
In particular, they were asking Kenneth Lay, the CEO at the time, questions about write-downs. Now everyone knows about the write-downs now, but back then it was just emerging news. Kenneth Lay didn't discuss write-downs in the presentation on the earnings calls, but there were six questions from analysts about write-downs in the Q&A section.
That showcased to us that Kenneth Lay was being reactive rather than proactive on that topic. Kenneth Lay didn't go and preemptively tell the market in that earnings call presentation about write-downs. He was being entirely reactive with the market on those. And then when he answered questions about the write-downs, he was being evasive.
He didn't actually answer the question about write-downs. He was pivoting, entirely off topic. So, there's two behaviors there that we identified from Enron. One was being proactive or reactive with information, and the second one was a very straightforward behavioral characteristic. When I answer a question, do I remain on topic to the question asked or do I go off topic?
And in finance, you know, both of those behavioral characteristics are important, and we've showcased in the research why they're important. That was the genesis of trying to identify these two behavioral characteristics from executive's calls at S&P Global Market Intelligence.
We have a product called Machine Readable Transcripts. That has earnings call transcripts going back almost 20 years, to 2006. It's approximately 250,000 machine-readable earnings calls.
So, we thought, “Okay, is there a way that we can systematically identify these two behaviors from executives across this massive corpus of earnings calls?”
And it turns out we were able to identify those two behavioral characteristics: proactiveness and being on-topic. And it turns out that when you look at them, proactive executives who voluntarily give information to the market -- rather than being asked about it -- outperform their reactive peers.
And on-topic executives – those executives who simply answer the question and remain on-topic to the question -- outperform their off-topic peers.
Dana Gardner: That's a very powerful result from an existing data set. You've been able to exercise new analysis on archives. And perhaps we'll get into this a little later, you can then help reinforce a good behavior from a bad behavior.
Tell us about S&P Global Market Intelligence and how you're using data science to create business services like this. And what's your role there?
Liam Hynes: S&P Market Intelligence is essentially the data and analytical service provider to financial institutions, academics, government institutions, and corporations.
We sell data and analytical tools. I sit on a team called Quantitative Research and Solutions (QRS) that sits within market intelligence. And we're the closest thing you can get to a client at S&P Marketing Intelligence.
We look across the vast array of data sets that we have. We essentially try and extract the value from those data sets, and then we articulate that value to our clients through research reports, to coding notebooks, coding blueprints, or to derived data products. So that's how we extract the value from our data and then pass it on.
Dana Gardner: And why is now an important and interesting time? What's coming together in terms of the technology, the price points of acquiring and using the technology, and the availability of the data? Why are we in a unique position to start offering new services like you've been describing?
Liam Hynes: We're in an inflection period when it comes to technological innovation, right?
Large language models (LLMs) have come into the fray over the past couple of years. We now have this new, shiny, amazing tool that we can point to all of our textual data, and essentially that's why there's been such an uptick in interest in these LLMs like ChatGPT and Llama.
They have trained themselves on all publicly available texts on the internet. But if you think about it, that's just the first level of analysis that the LLMs can do. I can go onto ChatGPT and do a multitude of things that save me time versus, let's say, jumping straight onto the internet.
But the next wave is textual information that's behind a paywall or is very difficult to collect. As an example, if I wanted to point a LLM to analyze those 250,000 earnings calls I was talking about, the LLM would have to go and first find all of the transcripts for each of those calls.
It would have to identify what was the presentation section, what was the question-and-answer section. At S&P Global Marketing Intelligence we have an agent, a machine-readable transcript product with all of the metadata associated with it.
So, I know the executives that are speaking. I know the analysts asking questions on the call. I can map the analysts back to their recommendations. I can map the company back to its financials. So, now we can unleash these LLMs on financial text to analyze the words that the executives and the analysts articulated on the call and identify if there are any signals embedded in this financial projection information.
It's opened up a whole box of really interesting research that you can do around financial information to determine if there's any alpha signals in there that we can bring back to our clients and say, “Here's how you go and calculate around these behaviors.”
We can containerize them and deliver the signals directly to our clients. Or we can actually just drop a coding notebook on them and say, “Here you go. Here's how you can do it yourself.” So, it’s a very interesting time to be in.
Dana Gardner: Yes. And you're looking at this through both structured and unstructured information. You're looking at the core data and the metadata, and you're also comparing it to other data sources in order to say, “Well, what differences can we discern between how a behavior took place in an earnings call, and then how the company performed over a period of time.”
When we do this right, when we can take advantage of these LLMs and data sources across both structured and unstructured sources, what sort of paybacks have you been able to get? What's the bottom line, so to speak?
Liam Hynes: Yeah, that's a good point, Dana. You have transcripts sitting in one corner. You have pricing information for the company sitting in another. You have the financials of the company sitting in another area.
So, one of the benefits of using Snowflake for this analysis is that all of S&P Global Market Intelligence data sets are available on the Snowflake Data Cloud. First, you have all of these very important company-specific data points in one place. I can go in and use the LLM to analyze the text, but I know then that this text was from an earnings call that was on April 1, 2025. I also have the pricing information from April 1, 2025. I also have the company's first quarter results.
So, you're able to create this data infrastructure where you have all of the important information in one place. And if I derive a signal from a LLM, I can then map that back to the company's financials and I can map it back to the company's points.
I'm able to understand if there's any behaviors or signals that the executive has spoken out on the call. I can then see if that correlates with an increase or a decrease in the price movement.
Dana Gardner: And what have you found when you do this comparison? What sort of results are you getting in terms of saying, “Ah, we now see something we didn't see before?”
Liam Hynes: When we look at the two behavioral characteristics that we identified -- and, again, I can go into the weeds later on on how we built those two signals -- we look at a specific executive. Essentially, we come up with a score on whether or not the executive is a proactive executive, or whether or not they're a reactive executive.
Let's say I have an earnings call that happens on March 15. I now have a score that says, “Okay, this executive was proactive.” Let's say there's another call that happens on April 15, and it's an executive who is reactive.
What we do at the end of every month is we identify every executive that had an earnings call in that month. If the executives were proactive on that earnings call and if they were in the top 20 percent of proactive executives, we build a long portfolio of those companies. And then on the short side, we would build a short portfolio of executives who are reactive.
Let's say it's the S&P 500 at the end of April, we have 100 executives who are proactive and 100 executives who are reactive, 20 percent of each. And we hold that long portfolio until the next month and then we re-sample it. And the same with the short portfolio.
When you do that over a 17-year period, rebalancing at the end of every month, proactive executives, or firms with proactive executives, outperform the reactive peers. They generate something like 250 or 300 basis points of pure alpha per year.
Now, that's one signal that we identified. The second signal that we identified was an on-topic alignment. So, when the executive answers a question and they remain on topic. Again, we go long. The top 20 percent of on-topic executives, and we short the bottom 20 percent. Again, that generates around 350 basis points of alpha per year.
So, without even looking at financials, without even analyzing any other information about the companies, just by using those two signals you're able to identify companies that outperform and underperform.
Dana Gardner: That's fascinating and very valuable. You can make inferences about how a company will perform based on how these executives themselves perform.
But it seems to me that over time you're also creating essentially a score for trust and credibility from the interview performance. And when you are able to then return this information in a feedback loop to those executives, they perhaps can improve how they communicate.
And that's where we started our conversation. Have we gotten to the point where we are able to take this analysis, using this great data across multiple sources, and apply them to bring it back to the person and say, “Aha, AI is going to help you be a better executive.”
Liam Hynes: Absolutely. If you think about it, there's multiple use cases, but the two main use cases here are for asset managers who want new signals to be able to identify companies that are outperforming or underperforming to build stock portfolios.
But then there's also corporates, right? So, if I'm a CEO of a corporation, I now have some valuable information. I know that if the market is looking for some information, it benefits me to be proactive with that information and give it to the market, rather for them look for it.
And the second component, very straightforward. If I'm an executive and I don't answer a question or remain on topic, I'm penalized for it. So, we already have a piece of software that we've built for investor relations departments at S&P Global Market Intelligence where they can pipe in their prepared remarks and we export the scores to them.
So even before an earnings call happens, executives are now prepping themselves and making sure that they're ready and that they're proactive. They're able to look at analyst questions that have happened, or they know that there's going to be certain analysts that on the call and they look at previous calls and see that they're asking about certain topics.
That means that, “Okay, if I'm an executive and I see last week one of my CEO peers was asked five questions around tariffs, for instance, I know that I probably have to be proactive with my information around tariffs on that call and proactively give that information to the market.”
I know that I'll be rewarded for that. And the second component is I can now prep myself specifically for those questions. The idea with the earnings calls is that the pre-prepared remarks are heavily scripted.
The CEO writes that script, but it is a heavily vetted document that the investor relations department, communications department, marketing, and legal all react to. Essentially the CEO is an actor. They're neutral, they're reading a script, and they're giving a presentation. What's in that presentation clearly matters.
So, executives are definitely spending a lot more time making sure that that presentation is the best that it can be. And then the second component is that they're also prepping themselves to be much more well prepared for the Q-and-A section.
More and more we see executives now going in to identify the analyst questions that were asked on previous calls and prepping and making sure that they are able to remain on topic to those questions that they're being asked.
Dana Gardner: And I should think that on the flip side, the people who are asking the questions, if they could avail themselves of this research, they might be able to come up with better questions or, or put them forward in such a way that they'll get more reliable results in terms of how this company's going to perform in the future.
And so that strikes me, Liam, as a very valuable process. Almost any company right now that's trying to grapple with how to monetize and benefit from AI, they can look for existing data and ways to share that data with people that will help them in their jobs, and then monetize that as a service.
It seems to me that that's a rinse-and-repeat sort of benefit that AI has with direct, demonstrable, and significant business benefits.
Liam Hynes: Absolutely. If you think about it, we've shown in the research that firms are rewarded and have higher stock prices if their executives are proactive and on topic, versus they're being reactive and off topic.
So now, you know there's a tool now that essentially can up the game for the executives, right? Executives know that they need to be much more transparent. They know that there are algorithms that are watching and listening to the call and identifying if they're going off topic. So essentially, it's raised the bar for executives across the globe are going to have to up their game when it comes to proactiveness and being on topic.
Dana Gardner: And they don't have to guess. They have data and they have inference, and they have science behind that saying, “Here's how you should behave. Here's how you go about these questions, and here's how you're going to get the best results.” So, less gut instinct and much more of a data-driven approach.
Liam Hynes: Yes, absolutely. And one of the things about that software that I mentioned earlier is it helps the executives write their presentation, right? And so, it not only scores them on proactiveness, but it also scores them on language complexity. Like am I using overly complex language?
Some previous research that we've done identified managers who use overly complex language underperform managers who use much more straightforward and simple language. You can also look at things like numerical transparency. When an executive is issuing a statement, if they accompany that statement with numerical facts, it has a lot more weight than a statement without numerical facts.
And another analogy I like to use is if I'm a football manager and I'm getting analyzed by a reporter at the end of the game, and let's say football manager is asked about their striker, and he says, “When our striker plays more than average games per season, he has the highest kilometers run on the pitch and scores more than average goals per game.”
And the second manager says, “You know, our striker is in the 97 percentiles for time on the pitch, and on average runs 11 kilometers and scores 1.2 goals per game.” The second statement holds a lot more weight than the first statement because it's accompanied by facts. So there are these other components that can aid the executives in their prep to make sure that they're delivering concise and succinct messages to the audience.
And one other thing about that software as well, we haven't embedded it into it yet, but phase two is where we're going to say, “How do we prep them for the Q&A?” What we're going to do is we're going to feed all of the previous analyst questions from that company and that company's peers into the LLM.
And then we're going to ask it to come up with hypothetical questions for the executives based on previous questions of its peers. So, it's almost like they'll be able to prep for the Q&A on the fly with relevant information.
Dana Gardner: Sure. The tool will be able to predict most likely the types of questions. And that way you can advance your preparations even more so.
And while this works for finance and perhaps sports, it seems to me that this is a function that you can take to almost any instance where you have important communications that you want to refine and improve. You have the data; you have the science to analyze it.
Let's dig into the science a bit, Liam, what is it underneath the covers? What's the secret sauce that's allowing you to do this? And what is the partnerships, the stack, and the underlying infrastructure that's come together at this auspicious time to enable you to do this fairly quickly and straightforwardly?
Liam Hynes: Sure. I'll start with the second bit first. We use Snowflake Cortex AI platform. All of S&P Global Market Intelligence data sets are available on Snowflake. And the reason we wanted to use that infrastructure was you can essentially pick LLM APIs off a shelf from Snowflake, and there's other vector embedding tools and summarization tools that you can use in Snowflake.
The fact that we had all of that data sets that I was talking about earlier -- the textual data, the pricing data, and the financial data -- all available on Snowflake and then the easy ability forvector embeddings, LLMs, summarization tools. It just meant that we had a one-stop shop to be able to do this analysis. So that's one component that was quite powerful.
The tech stack that we used was, let's start with how do you identify if somebody's answered a question and remained on topic? Well, if they remain on topic, they're going to use similar concepts. Topics and language that was used in the question. It means that we have to look at the question, at the language, in the question. We have to look at language in the answer, and we have to compare them to see if they're using similar concepts and similar topics.
And we do that by taking the question and taking the answer pair. And we use a LLM vector embeddings to turn that textual data in the question and answer into numerical data. So, think about vector embeddings as just a zip code or a post code for text. And so now that we've got numerical data to the question and answer, we can now compare that numerical data.
And if you remember, high school trigonometry or secondary school trigonometry, when I look at two vectors, if I get the co-sign of those vectors, I can determine if they're similar or not. If I have two vectors that are exactly the same, the angle between them is going to be zero, and the co-sign of zero is one.
If I have a vector that is the exact same, it's going to have a co-sign score of one. So as an example, the only time you get a co-sign of score of one in a Q&A pair is if an analyst said, what's your guidance for the fourth quarter? And the CEO said, did you say what is our guidance for the fourth quarter?
That's when you've got a perfect match. You know, that rarely happens and when it does happen, you know, we fold that Q&A pair into the next Q&A pair to combine the two of them. What happens is that when you calculate a co-sign symbol score for the question vector and the answer vector, a manager that is on topic will have a high cosign score close to one, and a manager or an executive who is off topic will have a lower score.
We can essentially now have a score, a ratio, anywhere between zero and one. And what we do then is we just go long short that score, so managers with a high score, we're on topic and we build long portfolios from them. Managers with a low score are off topic and we build short portfolios.
Dana Gardner: Well, I can certainly see where a chief executive or a financial officer that had this tool available to them would want to take advantage of it and certainly be thinking that if my competition is doing this and I'm not, then I'm at a significant disadvantage. So, seems like it's a no brainer that you'd want to avail yourself of these services
Liam Hynes: Absolutely. If you, if you want to be ahead of the game, then you need to be a CEO that analyzes their own language and how they articulate results and the operations of the company, too.
But for measuring the proactiveness, it's a bit more technical. Essentially what we have to do is we have to identify the topics that the analysts are asking in their questions, and we have to determine were those topics mentioned in the pre-prepared remarks. Now, there's an old school natural language processing way that you can do that by, you know, counting up and identifying the topics in the question, seeing if it during the remarks, and coming up with a scoring mechanism.
But we trained an LLM or through prompt engineering to pretend it was an executive on an earnings call. And what we did is we fed that LLM executive the analyst questions, and we asked it, we said, “Pretend you're an executive on an earnings call and answer these insight analyst questions.”
Now the difference is that we rig fenced the LLM to only to be able to answer the questions from the pre hurdle mark, right? So that meant the LLM answered the question, the analyst question, but only use the text from the pre remarks to be able to answer that question. And then we did the same process, right?
We take the LLM answer, the original analyst question, and we compare them. We run the cosign similarities. So, if the LLM was on topic, it meant that topic must have been covered in the preprepared remarks, meaning that the executive was proactive, and if the LLM was off topic, it meant that it couldn't answer questions.
So that topic wasn't covered in the pre-prepared remarks. Meaning that the executives being reactive when they were answering that question
Dana Gardner: If this happened in real time, then these analysts asking questions could get that red flag and say, “Oh, I need to drill down on that question. The LLM says that this is a potential area for a deeper dive.”
Liam Hynes: Yes. If you had a sell-side analyst who was waiting on the call to answer their question, and they had something that was in real time to tell them that this topic wasn't covered in the pre-prepared remarks and might be important, they could potentially use their air time to ask that question as well.
Dana Gardner: It's fascinating how you can take this in different tangents. What comes next? You mentioned Snowflake Cortex. We see more and more agents coming on board these days as we have more agentic AI capabilities. Where might this lead to? Is this just scratching the surface, Liam?
Liam Hynes: Well, you know, there's an opportunity there for executives probably to set up some kind of sell-side analyst agent, LLM agent, right? You know, they would train that sell-side analyst to look at all questions that analysts have asked previously on their earnings calls. They could look at questions that analysts have asked on that company's peers and competitors.
And then they could essentially input all of those questions into an LLM. And then that LLMcould come up with hypothetical questions, potentially even based on their presentation. Like a real life agent to help them prep for earnings calls to make sure they're sharp, concise, and on topic on earnings calls.
Dana Gardner: So, the equivalent of an AI sparring partner that you could get in the ring with and have a go few rounds before you get out into the real world?
Liam Hynes: That's it. Yep, exactly. Prep yourself for those four calls a year. Very interesting. And I want to note as well the benefits of Snowflake Cortex when we did this work. I think we processed 192,000 earnings call transcripts, and there's approximately 20 to 30 analyst questions in each transcript.
That's 2.2 million questions. We gave 2.2 million questions to the LLM to answer. And then the LLM spells out 2.2 billion answers. So, to be able to systematically do that was a great benefit. And the fact that when you look at that much of a corpus of earnings calls to show that systematically proactiveness and on-topicness matter, there's statistical significance there that showcases that this isn't just random.
Proactive managers outperform reactive managers. On-topic managers outperform their off-topic peers. But interesting enough, we also tested those two signals mutually. We asked, “Okay, what happens when you have an executive that exhibits both of those characteristics?”
We came up with four communication styles. The first one was proactive and on-topic. These are executives who give the market what they want. They're proactive with information, and when the analysts ask questions, they're on-topic. We then had proactive and off-topic, reactive and on-topic.
And the flip side was reactive and off-topic managers. This is where communication is totally broken down. So, they're being reactive, and the analysts aren't getting the information they want in the presentation. And when executives are quizzed for that information, they're going off-topic.
It's almost like a double whammy. Executives aren't putting it in, maybe they're avoiding thesubject of the topic. And when they're pressed on that topic, they're again being evasive and they're avoiding it. What we've noticed is in the research, reactive and off-topic executives are significantly penalized versus their proactive and on-topic peers.
When you combine the two signals together, it actually is a much more powerful signal.
Dana Gardner: Well, it certainly sounds like a must-have tool for the busy executive who can move markets with their words -- or lack of words. A very interesting use case that opens up a whole new era of different types of tools across all sorts of different types of communication.
And, Liam, you're going to be presenting more detail about this at the Snowflake Summit this June at San Francisco. I'm sure that will be a well-attended event.
Liam Hynes: Yes, absolutely. We're really looking forward to that. So, I'm presenting on Monday, June 2. I have a 45-minute presentation. I'll be getting into the weeds on the economic rationale and how we constructed the signals and some of the back test results.
And then on Thursday, we have a hands-on lab. So if you think about it, I'm handling the theory on the Monday, and then there's a 19-minute hands-on lab on Thursday for any data scientists really want to get into the weeds and replicate our work.
Dana Gardner: I'm afraid we'll have to leave it there. Thanks so much to our latest Data Cloud Podcast guest, Liam Hynes, Global Head of New Product Development at S&P Global Market Intelligence. We really appreciate your sharing your thoughts and experience in this fascinating use case, Liam.
Liam Hynes: My pleasure, Dana. Thanks a million.
Sponsor: Snowflake.
No comments:
Post a Comment