today at 1:41 PM
My main fear is that the integrity of everything will be compromised, but everyone will keep acting as if it's not. I feel like it's already somewhat the case but I'm worried about it getting worse.
My prediction is that some people will be able to use ASI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would approach 0% but one asset is going to grow at a faster rate than all other assets (E.g. an AI company) and everyone is going to start piling their infinite money into that asset; every other asset will crash to $0 because of the rapid price acceleration of the top asset. Then fiat money would cease to exist and shares of the top AI company will be used as money instead.
today at 11:55 AM
If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
today at 1:25 PM
There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
today at 1:30 PM
> There are a huge number of people involved in AI outside of the major AI companies
How many, approximately?
today at 1:00 PM
An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.
today at 1:20 PM
> there's a deep sickness here which causes people to flee from the inherent value (relatively modest)
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
today at 1:19 PM
I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
today at 12:14 PM
The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
today at 12:19 PM
> If it weren’t US companies, it would be others.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
today at 12:56 PM
I remember reading about the development of neural networks in a magazine from the 1980s.
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
today at 1:01 PM
> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did
Uh? In what universe are openai and anthropic not shadow groups?
today at 1:09 PM
Are you misunderstanding on purpose? OpenAI and Anthropic let the public use their AI models. Shadow groups would mean that people wouldn't even know about the existence of AI.
today at 1:14 PM
> If it weren’t OAI and Anthropic, it would be others.
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
today at 1:07 PM
I’m sorry, but the AI workers are as much to blame.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
today at 11:18 AM
"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
today at 1:23 PM
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
today at 11:55 AM
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
today at 1:32 PM
If people's pensions & life savings are flushed out the back door (literally), rest assured attention WILL be paid.
today at 12:25 PM
Blockchain dapps were getting hacked long before AI. Not a signal
today at 12:15 PM
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
today at 12:25 PM
Would it? I don't think so. Not if the weapon is an indiscriminate one and makes everyone worse off. Yes, some people (maybe military generals) might support it. But there would be near universal hatred for it, if people wake up one day and find their bank accounts not working or their eletricity not working.
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
today at 12:57 PM
> Not if the weapon is an indiscriminate one and makes everyone worse off.
Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.
today at 1:02 PM
> Nuclear weapons are indiscriminate and make everyone worse off if used.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
today at 11:47 AM
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
today at 1:04 PM
It would be a lesser evil rather than good.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
today at 11:51 AM
> people will die.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
today at 12:57 PM
So Hinton doesn't trust the tech leaders but then the consequence of having governments censor LLMs is that only tech leaders and governments have unlimited access to AI.
If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.
today at 1:04 PM
I would posit that systems being hacked by AI were already vulnerable to the huge caches of secret zero days held by intelligence agencies and organized crime around the world. Which we know exist.
Insecure is insecure. These systems never were secure.
today at 11:43 AM
Of course, we are. Back in the late 80’s early 90’s, no one working on ML was aspiring to develop systems which would be used to generate the kind of AI crap which plagues us today.The good is being smothered by the absolutely soul-crushing pointless.
today at 1:09 PM
AI exposes the Sturgeon's law (90% of everything is crap), and amplifies it much more, and anything good is drowned out. Most of human-generated popular music is slop. Most of human generated code is buggy, unmaintainable and full of debt. Social media and youtube have always been dominated by click-bait and mediocre time-wasting crap.
today at 12:23 PM
I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.
today at 1:06 PM
Before AI I was already being smothered by soul crushing pointless junk made by humans.
You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.
today at 12:08 PM
"We". Who "is" we? This isn't about you. or me. there was never a we. Before you open your mouth or put your pen down, wouldn't you think of the shareholder value for once?
today at 12:12 PM
Speak for yourself - I'm in AI heaven.
Programming has never been this awesome.
today at 1:33 PM
The people in wall-e were in AI heaven.
today at 12:13 PM
“I think [Mark] Zuckerberg and [Elon] Musk and Sam Altman, they understand the risks, and they don’t give a shit. They’d rather have the money. That’s my belief.”
today at 12:58 PM
Such a smart guy so late for the party. Thanks to him and others who understand a lot about tech, we gave monster a weapon to destroy us.
I still don't know how Karpathy still works with those folks. I really think the guy is great but he makes me wonder
today at 12:06 PM
As is the case with any technology in the hands of careless people, but unfortunately we can't wait for society to catch up morally to the technical standards, innovation is organic and breakthroughs have no timing.
today at 12:29 PM
"It might turn out we were better off not opening Pandora's box" says man who helped open Pandora's box.
That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.
today at 10:54 AM
AI winter has arrived.
today at 11:16 AM
Said AI "experts" for the last 10 years.
today at 12:18 PM
Nobody has said that. The winter will probably come, but it's not at all clear when. First the bubble must burst.
today at 1:00 PM
today at 1:07 PM
almost every story that has ever been written about a world with AI has posited that humanity was better off without AI.
today at 12:19 PM
I think we did good inventing it and it's inevitable. Lets use it well.
today at 11:05 AM
modern day Mary Shelley
today at 12:05 PM
To be honest, I think it depends on the environment. In the past, if you didn't come out of a good university, your access to knowledge was limited. But these days, in terms of access to knowledge, I think things have clearly improved, whether or not AI is illegally sharing knowledge.
Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.
I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.
But considering what access to knowledge was like before AI came along, I think having AI is better.
today at 1:06 PM
The internet pre-AI did far more to make knowledge accessible than AI has done. Its useful, but its not the critical thing.
today at 11:09 AM
This is a bit like Rudolf Diesel saying maybe we're better off without airplanes. The field has long since left this guy behind.
today at 12:30 PM
If the resulting issues are correct it doesn't matter whether the field left the guy behind or not.
He would still be right and the field practioners would be wrong.
today at 12:01 PM
So pumping CO2 into the atmosphere is a good thing and if someone does a backflip by recognising their mistakes that is a bad thing?
today at 12:19 PM
We should regulate data centers to require fully renewable power (solar/wind/hydro + battery).
today at 1:23 PM
This is tangential encouraged by your point and other forays here. But one does wonder what arguments politicians (the power lies with them) understand when they make their pronouncements. The UK has for instance, prior to the new administration, totally and vociferously supported net zero. From the point of view (a fact for many but not for all informed authorities) that climate change is caused by CO2 and we are headed for a disaster - Net Zero is a logical step to trying to save humanity and the whole ecosystem. Now (November) I read that the UK powers that be, appear to have put it aside in actions at least. Countdown to Net Zero doesn't matter any more; open the oil fields, get rid of subsidies for photovoltaic and wind - and other measures. My point is not a political jibe but a stance I suspect to apply in other areas. Is climate change going to be a catastrophe or not? They should make up their minds.
today at 11:19 AM
I guess capsule network haven't improved ...
today at 1:06 PM
I actually laughed out loud.
I do find capsules interesting though.
today at 12:56 PM
What is the point of this quote? Or any of his quotes for that matter.
today at 11:54 AM
We should introduce regulation so only credentialed software engineers are allowed to use AI.
today at 12:42 PM
I would rather see the onus on the company owners to be responsible for what they sell. Serious penalties for seriously bad products.
today at 1:16 PM
Absolutely! And software engineer should be an hereditary title as well.
today at 12:10 PM
In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.
Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.
The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.
But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good
today at 12:08 PM
It might turn out we were better off without nuclear weapons, but so far I’m happy we haven’t had another world war (in the traditional “total war” sense).
No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.
Same with AI. Thus far we’re better off with it than without it. That may change.
I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.
Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?
(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)
today at 12:32 PM
>so far I’m happy we haven’t had another world war
The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.
today at 1:40 PM
> roughly half the world each.
Both the USSR and US fought multiple wars directly against the proxies of the other. They weren't simply hanging back and letting others do the dirty work. In most any other era one of those would have become a full blown war between the two. Both sides pushed the line of not directly involved and then both sides looked the other way versus escalating. Nukes made the alternative too much of a risk for both sides.
today at 11:24 AM
oh word?
today at 12:10 PM
[flagged]
today at 11:15 AM
[flagged]
today at 10:32 AM
The scene is bleak, silent, desert-like with a few dead trees bleached white by unfiltered UV. Nothing moves. In the distance a cave, and from that cave
... no shit Sherlock
today at 10:54 AM
9 minutes. Was that enough time to read the article?
today at 12:23 PM
My reader mode says 6–7 minutes, so, yes.
TFA could have been two sentences:
He warns that AI might cause serious problems: power failures, job losses, and greater social inequality.
He believes AI might improve our lives, but governments must make sure it is safe.
today at 10:56 AM
Did you look at it? It wasn’t that long.
today at 11:01 AM
I personally feel reading the article before commenting affects the purity of my vibe before posting
today at 11:05 AM
vibe comments!
today at 10:36 AM
We should have not left the trees. You cannot disrupt banking and power if they do not exist. Checkmate!
today at 11:50 AM
The ghost of Douglas Adams has entered the chat
today at 12:05 PM
So why did he start to develop it in the 1970s?
today at 12:17 PM
Why did he start a research direction that 50 years later "might turn out" to not be a good idea?
That's a relatively silly question, don't you think?
today at 12:26 PM
No it's a completely serious question. The goal has always been to replicate human/biological thinking processes in software, and now that this has been more or less achieved he's suddenly not happy about it. There's been plenty of opportunity to actually think about "what if all of this actually works" but they chose not to.
today at 12:27 PM
Sounds like sour grapes. He worked on it for a long time but he doesn't own a trillion dollar company. Missed the boat?
today at 11:06 AM
We are not but we need a better approach, https://www.ethicalacceleration.org
today at 11:20 AM
> It could help cure diseases, end boring work and give us more free time.
Seriously? It seems to be causing more problems, giving us more boring work, and less free time.... sounds like absolute bullshit.
today at 11:49 AM
"Could" does a lot of work in these kinds of claims.
We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.
The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.
today at 11:52 AM
> We need to build it as fast as possible because it could cure all disease
Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.
today at 11:55 AM
How has AI given you less free time?
today at 12:01 PM
I have to spend extra time just trying to find AI-free things, AI-free writing, AI-free communities, and people who aren't addicted to AI. And get noticed/get a job by having to push through all this AI static. And having to interact with AI-based customer service agents that take 10x as long to resolve the problem as a human being. And having to wait a bit longer because there are 10x more 'Are You Human' Captchas slowing down website loading. And having to download OS updates that are 2x as large because they contain AI models. And having to double-check everything because there's so much more hallucinated crap out there.
The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.
In short, it takes a LOT more time just to find some sanity now.
AI is shit.
today at 12:26 PM
That's the whole point of the site. Did you read it. AI should not cause harm, direct or indirect.
today at 12:37 PM
Of course I read it. But my thesis is that one cannot avoid that. Causing harm, whether direct or indirect, is the fundamental nature of AI.
today at 1:32 PM
It can be avoided, humans control AI still. Humans are the ones causing harm with the help of humans.
today at 12:13 PM
I mean, it could if "we" "just" got out of globalized financial capitalism, changed everything about our political, economical, financial, social systems, etc.
Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way
today at 12:15 PM
I don't think accelerationists think we can do anything. The thought process of the accelerationist is partly fueled by the 'excitement' of betting and playing musical chairs and hoping to be the first ones out holding a bag of money and partly fueld by the delusion that technological advancement is always good at any level of complexity.
today at 12:20 PM
today at 11:50 AM
You’re a cultist
today at 12:25 PM
How did you come to that conclusion?