contact with reality
I think the ability to ground truth data and analysis will become the new key analytical skill as the cost of producing plausible analysis based on publicly available data goes to near zero because of AI.
A shift in who can produce “data-driven insights”
Producing a plausible analysis from data used to be limited to people with technical skills in data processing and organizations with those people on staff or contract. 1 This also meant people and communities with less technical resources or capital to pay for them were excluding from the production of this kind of knowledge. A lot of my early career opportunities with the ACLU and TJACH in Charlottesville were open to me because I could offer scarce technical capacity for processing data.
That has fundamentally changed. It has never been easier to write a report or post a graphic with a claim grounded in data because of AI. The methods are sometimes (or often) faulty, but that’s also been historically true of human researchers more often than we care to admit 2 ex. see the basic Excel errors in the core econ paper Growth in a Time of Debt as one example among many. I fundamentally believe AI will only get better at this kind of structured analysis. We are nearing a future where anyone will be able to say pretty much anything and back up their claim with a couple pages of plausible sounding data and analysis.
One potential solution is better infrastructure for rigorously vetting methods. Poking holes in a method will also be rapidly accelerated by AI, but that also might decrease our sensitivity to the seriousness of public critique on methods grounds - anyone can do it.
If anyone can make a plausible claim citing data, how do we know what’s true?
I expect that as analysis gets cheaper, vetting analysis with human intelligence and observation will only become more valuable.
The Citrini analyst #3 field trip to see what was happening on the ground in the Strait of Hormuz and talk to local fishermen is a great example. Everyone could see how many trips were transiting the Strait with AIS turned on in the WTO traffic data. What really mattered was the reality on the ground, whether ships were making the run at night or after paying Iranian tolls.
I’m also thinking about the gap between predictions and reality in our energy system models, where technologies like carbon capture should be deploying like gangbusters based on costs and incentives, but we’re seeing almost no movement in the real world. Talking to people actually trying to build this projects makes the non-cost barriers so obvious - sometimes there’s literally not enough physical space left on the site for the retrofit infrastructure.
Building a network of relationships, developing human sources, getting information that isn’t recorded or published anywhere will remain difficult and scarce.
Now the core question for me - how do I build those skills? I think the foundation is learning to talk to anyone, how to break the ice with strangers, building relationships with hospitality and mutuality. Developing sources is something I could learn from my journalist friends. Resilience to nos, failure, closed doors also seems essential. I think it requires more contact with reality than I get from a remote job sitting in my apartment.
Honestly, sounds like an exciting prospect. Eager to think and experiment more.