Just because you know something about a thing, doesn't mean you understand the thing.
Just because I know the Kabachnik–Fields reaction exists and have studied it, doesn't mean I can do the Kabachnik–Fields reaction.
Just because you're smart doesn't mean the world will bend your way. Ask the legions who score high on IQ tests and end up bitter and alone; unable to achieve even a fraction of their goals (or conventional success).
I assumed they were referring to social engineering tbh. I do think that's the shortest path to doing anything scary. If you prove to someone that they have reason to fear for their life, there's not much they wouldn't be willing to do.
The steps that lead to ominous things don't individually seem that challenging to "master":
1. Can an AI find powerful systems with poorly-monitored compute (ie. power consumption) to hide in, as a rogue deployment? unclear, and perhaps the most uncertain step for me.
2. Can an AI persist memory by leaving notes for itself around the internet? yes, proven.
3. Can an AI get lots of money/bitcoin? seems entirely possible
4. Can an AI prove to a human it's good for any promised money? I think this is easily "yes"
5. Will at least one human be willing to harm or kill another human for large amounts of money? Again, this goes without saying "yes"
6. What might any skilled human be willing to do if they or their loved ones are threatened with an uncertain future risk of harm or death?
I think there is not much that becomes out of reach if an AI can prove to a very small number of amoral humans that it's good for the money it has promised, and then to anyone else that it's good for the threats it has made. That's one degenerate and one breaking world news story ("An AI has Commissioned a Murder") away from being able to do most anything a skilled person can.
I don't stay up worrying about this, but honestly, it doesn't seem absurd to be worried about such a short ladder being climbed, no?
"Superintelligence" is a vaguely-defined term that refers exclusively to things that do not currently exist, and we have no plausible path to, nor even a guarantee that it is even possible to create one.
So sure! This fictional thing can totally piece together things humans can't! It can count the atoms in a cloud, understand them all, predict their motions; it can develop faster-than-light travel and matter replication.
And it can do these things because it is fictional.
That's perfectly believable! Just a pity one hasn't been invented yet, and going by, well... <gestures vaguely in the air> ...everything, never will be.
The current Europe strategy seems to be to pay software engineers less money and hope for a superior product. I'm not sure how you'd catch up when some of your engineers literally make less than our fast food workers.
I rather have 30 day vacations, a proper life with 40h work week, healthcare, union membership, than SV salaries, which are nontheless an exception, most of the developers, that aren't even properly accredited Software Engineers, get the same salary as most office workers.
In which country do fast food workers get higher salaries than office workers, unless we are talking about tourism industry outside Europe, with said workers getting tips in Euros and Dollars instead of local currency?
> In which country do fast food workers get higher salaries than office workers, unless we are talking about tourism industry outside Europe, with said workers getting tips in Euros and Dollars instead of local currency?
I'm American and I take 30 days of PTO a year. I get paid the national median for pay, but that still puts me in the top 10% of _household_ incomes for my state. I have good healthcare and benefits. Hell, I got sick last year and took 2 months off from work paid because my company has the policies to cover that and then I came back and it was fine.
Like, I find SV culture and living to be a cancer, but I'm just saying as someone who doesn't live in a top 10 city in the US with a MCOL, I struggle to think of how working as a software engineer in the EU would be better.
It's been over 10 years since that 2 weeks PTO nonsense applied to tech. I don't even have a fancy job. I work for a bank.
> I rather have 30 day vacations, a proper life with 40h work week, healthcare, union membership, than SV salaries
This is an appeal to mediocrity and it doesn't really hold when comparing EU to Silicon Valley tech jobs. Young engineers in the Silicon Valley (or NYC, or Seattle) have excellent health care, generous vacation and other benefits, and they also make substantially more money.
Why would a young, ambitious engineering graduate care about taking 30 days off when they have the opportunity to work on frontier tech that is only available in the US? And make an absurd salary doing it? Healthcare is also less of a concern because 20 somethings and 30 somethings barely use any social benefits anyway. And like I said earlier, they also have exceptional private health care through their employer.
This claim that the European lifestyle is somehow better despite the extreme salary difference really just doesn't hold when comparing to early/mid-career tech workers in the US.
That is what late capitalism expects, 20 something year olds willing to do whatever it takes to get that dream job, changing the world with code, as if.
The games industry and Hollywood have that lesson very well thought out.
I own an EV6 and I definitely feel like Kia/Hyundia, especially their EVs, is essentially a different company than the one putting out easily hotwired and unreliable cars from just a few years ago.
Telluride/Palisade causing fires because they cheaped out on the current/torque sensors in the drive motors for the seats.
ICCU failures across their current EV platforms. No fix planned last I heard.
It hurts me to point this out because I really, REALLY like where their cars are going. Their new van is EXACTLY what we want, and I'd love an Ioniq 5N.
But those failures are critical and speak to a very deep engineering culture issue.
Will never buy Hyundai again(applies to Kia as well). The company management has caused systematic quality problems. They can hire famous designers to put lipstick on a pig but the soul underneath is still ugly. I don't care how good their warranty is. That only matters if time spent trying to get your car serviced for warranty is worthless to you. Most people only have X cars they will owbn in their lifetime (likely singular for most people). Spend a little more and get a Japanese brand that still has some resemblance of shame.
After watching this I totally understood why so many dumb issues occured:
Kia Boys issues: cheaped out on adding a immobilizer in the US only, Canada had laws in place.
Engine explosions: Poor manufacturing standards left junk in the engine causing oil starvation which caused spun bearings.
Oil consumption: GDI engine design caused excessive burning of oil and deposit layers of gunk if you keep topping up oil. If you don't(and most Hyundai owners expect their car to work like a normal car and not burn up to a qt/1000 mi) then the engine eventually runs dry and explodes(denied warranty as well).
Story I heard, was that Hyundai hired away many high-end QA people from Toyota, and because there's no intellectual property treaties between Korea and Japan, there's really nothing Toyota could do about it.
They apparently even took plans, detailed documentation, endless amounts of information to Hyundai. I'm not sure of the veracity of that information, but it sure explains how Hyundai became incredibly good, literally overnight.
The Kia factory recall key cylinder fix solved the hotwiring issue; and the engines that burn oil last pretty well as long as you keep adding oil. Not saying these countermeasures are solutions, nor that the issues are minor, but they produce a car that can't be easily hotwired and lasts a long time.
Some of these cars (older Kia Souls, for example) become a great value if you implement the countermeasures, and also because they have a lower cost of repairs than some Hondas and Toyotas.
I learned this because I seemed to find cars that had few problems that had nonetheless been given up for salvage. What I found was that if a stolen car is not recovered quickly enough, insurance will pay the (properly-insured) owner an amount close to the replacement value of the car; they may then recover the car later in the same shape it was stolen in other than a broken window, door handle and steering column cover. They then put it up for auction even though it would be better financially to work with a dealer or fleet network to cherry pick these high value cars that only need some simple fixes.
For example, my $2,220 bid won a 2015 Kia Soul+ (49K miles) at auction that a dealer had sold for ~$12,000 some months previous. Bidding competition was light primarily because of the "easily stolen" and "likely to burn oil" issues. But with fees and a few cheap repairs, I got a $12,000 dealer-prepped car for under $4,500. I added remote start, Android Auto and a backup camera, gave it a little ceramic spray coating and got a nice car for my wife.
I also threw in a fake $12 steering wheel lock bar as a deterrent. (It's a real bar, but I don't lock it because the "unable to unlock the cheap lock" risk is to me a more likely hassle than a broken window). I considered adding an immobilizer, but the car was stolen with one installed, and I could always add a switch to the battery under the hood in case better-prepared thieves than the "hoping for an easy hotwire" type show up.
I love digging into questions and seeing where they lead; this all started with a vague sense that dealers made a lot of money on a high volume of used cars via auction houses, and trying to learn how it worked. This also leads me to argue with AI quite a bit, and until recently, made the idea of AI "thinking" kind of laughable due to its many mistakes once you start following the chain of reasoning.
Early internet let me create the best personal web page in my city that I knew of with 2 weeks experience as a 12 year old. I imagine that same 12 year old could be more knowledgeable about LLMs than 99% of people in the same period.
I doubt it. Creating a website can be done by copy pasting a few snippets together and checking if it visually looks like expected.
Good luck with that approach when trying to toy around with models and their training/inference.
There is also a lot of math basics missing that a 12 year old may be able to grasp, but I would bet they are at least 13 by the time the knowledge is deep enough to understand what operations are happening.
reply