I'm not sure who's saying that pandemic prevention helps against AI but I think they have a wacky threat model. AI just kills you by killing you? Expecting pandemic prevention to save you from AIs using pandemics is like expecting gun control to reduce AI risk because it stops AI from using guns - if you block off some specific strategy, it's allowed to use a different one that works!
Like a doomer with a non-wacky threat model, I think pandemic prevention is good because it prevents pandemics.
I think that it's more or less the official open philanthropy view that 1. The pandemics which are likely to kill enough people to pass their bar for importance are overwhelmingly likely to be engineered pandemics 2. AI will make it easier to create engineered pandemics even if it is not an autonomous agent. I am not privy to the full details of their reasoning about the relevance of pandemic prevention to autonomous superintelligent AI.
Very glad someone put this into words, much better than I could've. I think of Dorothea's plight in Middlemarch -- EAs have a longer history than people think, it goes back to Mother Theresa, back to the early Christians. There will always be people who fervidly pursue the good and the beautiful, and it's sad to see doom descending on them.
in my understanding the error in the proof is that square roots have two results: a positive and negative one. i/1 is the positive one, and 1/i is the negative one. So line 4 is incorrect (equating the square roots).
I really like this piece despite disagreeing on the content. I think people writing well-formed/high-quality counterarguments or countersentiments to dominant norms and ideas is really important so there is some check on things like groupthink. Good job :) <3
i work on ai but i feel you. ea is in part about not trusting your intuition. buy fuzzies and utils separately, etc. ai is ugly in this sense, it's not fuzzy
hm maybe but then i am struggling to see how we should change our ways. less certainty in the singularity narrative? less consequentialism to not be like the guy who saves the ea over non-ea? more diversification of the portfolio (iirc openphil was like 50/50 x-risk vs other stuff last time i checked)? being less annoying communicators (ig connects to the first point)?
I dunno man. I have a broad, not overly legible sense that This Is All Fucked, in a real important way that will lead to worse outcomes, but I don't know how to fix it. If I had solutions I'd be posting those instead. But TBC I don't think the mistake the ea-over-three-babies guy was making was being Too Consequentialist. Its definitely not a mere failure in communication either! (I guess my objection is most legible when it comes to the singularity narrative if I had to pick something.)
i still don't fully understand how This is All Fucked, but i appreciate the attempt to explain the mood! however, even if this is not about fuzzies or aesthetics, relying on intuition in this way still seems antithetical to the ea approach, which is at the core about following the spreadsheets and the arguments
Well not to uno reverse card you on rationalist precepts but the primary thing when you take a sword into your hands is to cut the enemy. There are some situations where intuitions are bad and spreadsheets are good--global health and animal welfare are cases where you can arrive at true, surprising, and actionable conclusions by using spreadsheets and arguments. But I think eg malaria nets and chicken factory farms are unusually well defined and well scoped areas--exceptions rather than rules. Simply because you have used a spreadsheet and made an argument doesn't guarantee that you will cut the enemy. There are myriad ways to be wrong about The Good with spreadsheets and arguments. It’s sort of neither here nor there imo.
so is the argument then that ai is less susceptible to reasoning and spreadsheets and we should therefore rely more on vibes? i think this is empirically wrong, eg here
looks like spreadsheets (scaling laws) and arguments (crazy futurism of kokotajlo, buck and the like) actually did lead to surprising actionable conclusions!
I think we are talking about fundamentally different scales/levels of organization and therefore talking entirely past each other, so I have nothing else to say
so if i understand you correctly, you like the parts of ea that are pointed at the "beautiful," but thinking what is beautiful requires intuition. i agree that ai as a direction of work is not intuitively beautiful in the way in which ghd, animal welfare are. in that, we agree. i also think that rejecting intuition and correspondingly the notion of beauty is a core part of the ea doctrine, so ig in that we disagree
I think the kind of ugliness present in the EA AI scene is actually quite important to pay attention to, and is not at all mapping onto the "fuzzies vs utils" distinction. Traditionally EA work in ghd and animal welfare is Also not especially "fuzzy"! I think the ugly qualities of EA AI work/culture/epistemics/etc as it exists are Actually Bad and very much *not* a consequence of a mere aesthetic judgement or a perceived lack of Fuzziness. I think insofar as the stated precepts of EA tempt one to ignore the various signs of Badness present here because "ah this is just what it looks like to be Actually Effective in the purchase of utils, nothing wrong here" , this is a profound flaw within EA
If you think of AI as our descendants, it takes most of the sting out of the thought of being replaced. If you remember that they're superintelligent, the paperclip problem goes away. If you consider ur-AI's strategy for replacing us, making our lives so engaging, interesting, and fun that the fertility rate has already fallen below replacement for the rich half of humanity, the possibilities of Terminators and plagues seem quaint and silly.
I'm not sure who's saying that pandemic prevention helps against AI but I think they have a wacky threat model. AI just kills you by killing you? Expecting pandemic prevention to save you from AIs using pandemics is like expecting gun control to reduce AI risk because it stops AI from using guns - if you block off some specific strategy, it's allowed to use a different one that works!
Like a doomer with a non-wacky threat model, I think pandemic prevention is good because it prevents pandemics.
I think that it's more or less the official open philanthropy view that 1. The pandemics which are likely to kill enough people to pass their bar for importance are overwhelmingly likely to be engineered pandemics 2. AI will make it easier to create engineered pandemics even if it is not an autonomous agent. I am not privy to the full details of their reasoning about the relevance of pandemic prevention to autonomous superintelligent AI.
Very glad someone put this into words, much better than I could've. I think of Dorothea's plight in Middlemarch -- EAs have a longer history than people think, it goes back to Mother Theresa, back to the early Christians. There will always be people who fervidly pursue the good and the beautiful, and it's sad to see doom descending on them.
in my understanding the error in the proof is that square roots have two results: a positive and negative one. i/1 is the positive one, and 1/i is the negative one. So line 4 is incorrect (equating the square roots).
Yes, you have to be careful taking the square root of both sides for this reason. You can see it in the simpler example
1 = 1
(1)^2 = (-1)^2
sqrt((1)^2) = sqrt((-1)^2)
1 = -1
Tell me more about the true and the good and the beautiful. Aren’t they in an LLM?
> Back in 2022 (what some call EA’s summer of love and what I call EA’s deranged cokehead era)
Wow, great epithets. I didn't know them. I usually call it the fever dream (usually in the context of waking up from it in November).
Beautiful post! My feelings about the matter are altogether very different, but I agree it's OK not to care about AI.
Fever dream is a good one yeah
I really like this piece despite disagreeing on the content. I think people writing well-formed/high-quality counterarguments or countersentiments to dominant norms and ideas is really important so there is some check on things like groupthink. Good job :) <3
i work on ai but i feel you. ea is in part about not trusting your intuition. buy fuzzies and utils separately, etc. ai is ugly in this sense, it's not fuzzy
hm maybe but then i am struggling to see how we should change our ways. less certainty in the singularity narrative? less consequentialism to not be like the guy who saves the ea over non-ea? more diversification of the portfolio (iirc openphil was like 50/50 x-risk vs other stuff last time i checked)? being less annoying communicators (ig connects to the first point)?
I dunno man. I have a broad, not overly legible sense that This Is All Fucked, in a real important way that will lead to worse outcomes, but I don't know how to fix it. If I had solutions I'd be posting those instead. But TBC I don't think the mistake the ea-over-three-babies guy was making was being Too Consequentialist. Its definitely not a mere failure in communication either! (I guess my objection is most legible when it comes to the singularity narrative if I had to pick something.)
i still don't fully understand how This is All Fucked, but i appreciate the attempt to explain the mood! however, even if this is not about fuzzies or aesthetics, relying on intuition in this way still seems antithetical to the ea approach, which is at the core about following the spreadsheets and the arguments
Well not to uno reverse card you on rationalist precepts but the primary thing when you take a sword into your hands is to cut the enemy. There are some situations where intuitions are bad and spreadsheets are good--global health and animal welfare are cases where you can arrive at true, surprising, and actionable conclusions by using spreadsheets and arguments. But I think eg malaria nets and chicken factory farms are unusually well defined and well scoped areas--exceptions rather than rules. Simply because you have used a spreadsheet and made an argument doesn't guarantee that you will cut the enemy. There are myriad ways to be wrong about The Good with spreadsheets and arguments. It’s sort of neither here nor there imo.
so is the argument then that ai is less susceptible to reasoning and spreadsheets and we should therefore rely more on vibes? i think this is empirically wrong, eg here
https://dylanmatthews.substack.com/p/the-ai-people-have-been-right-a-lot?r=22k2w5&utm_medium=ios
looks like spreadsheets (scaling laws) and arguments (crazy futurism of kokotajlo, buck and the like) actually did lead to surprising actionable conclusions!
I think we are talking about fundamentally different scales/levels of organization and therefore talking entirely past each other, so I have nothing else to say
?it sounds like we actually disagree then? at least implicitly
so if i understand you correctly, you like the parts of ea that are pointed at the "beautiful," but thinking what is beautiful requires intuition. i agree that ai as a direction of work is not intuitively beautiful in the way in which ghd, animal welfare are. in that, we agree. i also think that rejecting intuition and correspondingly the notion of beauty is a core part of the ea doctrine, so ig in that we disagree
I think the kind of ugliness present in the EA AI scene is actually quite important to pay attention to, and is not at all mapping onto the "fuzzies vs utils" distinction. Traditionally EA work in ghd and animal welfare is Also not especially "fuzzy"! I think the ugly qualities of EA AI work/culture/epistemics/etc as it exists are Actually Bad and very much *not* a consequence of a mere aesthetic judgement or a perceived lack of Fuzziness. I think insofar as the stated precepts of EA tempt one to ignore the various signs of Badness present here because "ah this is just what it looks like to be Actually Effective in the purchase of utils, nothing wrong here" , this is a profound flaw within EA
If you think of AI as our descendants, it takes most of the sting out of the thought of being replaced. If you remember that they're superintelligent, the paperclip problem goes away. If you consider ur-AI's strategy for replacing us, making our lives so engaging, interesting, and fun that the fertility rate has already fallen below replacement for the rich half of humanity, the possibilities of Terminators and plagues seem quaint and silly.