LLMs performed best on questions related to legal systems and social complexity, but they struggled significantly with topics such as discrimination and social mobility.
“The main takeaway from this study is that LLMs, while impressive, still lack the depth of understanding required for advanced history,” said del Rio-Chanona. “They’re great for basic facts, but when it comes to more nuanced, PhD-level historical inquiry, they’re not yet up to the task.”
Among the tested models, GPT-4 Turbo ranked highest with 46% accuracy, while Llama-3.1-8B scored the lowest at 33.6%.
It would be interesting to give these scores a bit of context: what level would a random person off the street, a history undergrad and a history professor score?
I think they all would have performed significantly better with a degree of context.
Trying to use a large language model like a database is simply A misapplication of the technology.
The real question is if you gave a human an entire library of history. Would they be able to identify relevant paragraphs based on a paragraph that only contains semantic information? The answer is probably not. This is the way that we need to be using these things.
Unfortunately companies like openai really want this to be the next Google because there’s so much money to be hired by selling this is a product to businesses who don’t care to roll more efficient solutions.