Adarti

Protest against Death

2012-01-12T14:06:00.001+02:00

Tomorrow there will be a demonstration for radical life extension and against death in Tel Aviv. Here it is on the Facebook Event page (Hebrew), IEEE Blog (English), and Singularity blog (Hebrew).

The protest will be near a cemetery in downtown Tel Aviv. A suitable venue, and the cemetery is defunct, so that mourners will not be offended.

I wrote a blog post in Hebrew explaining why death is bad :-) but there is enough material on the subject in English, so I won't go into that here.

It's nice to see that they got video greetings endorsing the protest from Frank Tipler, Natasha Vita-Moore, and Aubrey de Grey among others.

Unfortunately, an arts magazine, Haba Lehava, was also involved, and they were a little too artsy for their own good. They confused the fight for life with an artistic statement, and they mixed in ordinary political issues (public housing, etc.) and harmful nonsense like "Haba Lehava, the only magazine which promises its readers eternal life."

But hopefully the actual protest will stay focused.

I like the idea of clearly speaking out in favor of ongoing health and against aging. I think that though the idea is generally scoffed at, it is (in contrast to many other transhumanist ideas), not so far away from being accepted as common sense if said often enough and clearly enough.

אני מתנגד למוות

2012-01-11T17:23:00.003+02:00

אני תומך במחאה נגד המוות שתתקיים בתל אביב ביום ו', 13.1.2012.

אני רוצה שכל אחד שחפץ בכך יוכל להיות בריא היום, מחר, ולתמיד. אני רוצה שכל מי שחפץ בחיים לא ימות. אני רוצה שכל מי שמעדיף לא להזדקן יישאר עם גוף חזק ובריא. אני יודע שהדבר לא אפשרי היום--צריך לשאוף לכך בהתמדה.

טוב לראות שגייסו דברי תמיכה מנטשה ויטה-מור, יו"ר הארגון "אנושות פלוס", מאוברי דה גריי, חוקר מוביל במיגור ההזדקנות, ועוד.

כשאני אומר "אני מתנגד למוות", התגובה השגורה היא "אבל מוות לפעמים הוא דבר רצוי." כן, לפעמים גם כריתת גפים היא דבר רצוי, במקרה של נמק חמור. האם אנחנו אומרים ש"כריתת גפים היא דבר רצוי"? מוות הוא דבר רע. יש מקרים כאלה: "טוב מותי מחיי" (יונה ד', ושימו לב שיונה מדבר דברי רחמנות עצמית טפשיים) "וַתִּבְחַר מַחֲנָק נַפְשִׁי מָוֶת מֵעַצְמוֹתָי. מָאַסְתִּי לא לְעֹלָם אֶחְיֶה חֲדַל מִמֶּנִּי כִּי הֶבֶל יָמָי." (איוב ז'); "מְחַכִּים לַמָּוֶת וְאֵינֶנּוּ, וַיַּחְפְּרֻהוּ מִמַּטְמוֹנִים (איוב ג'); "וְנִבְחַר מָוֶת מֵחַיִּים" (ירמיהו ח'). הפיתרון הוא לא לבחור במוות, אלא לתקן ולרפא את המקרים הנוראיים האלה.

משיבים "אבל צריך למנוע פיצוץ אוכלוסין". בשנה 1650, אם היינו אומרים להורים צעירים "בעזרת היגיינה, חיסונים ותזונה, רוב ילדיכם יכולים להגיע לבגרות ולא למות.... אבל אוכלוסית העולם תעלה ממיליארד לשבעה מיליארד", מי היה עונה "אנא, תנו לבניי למות ממחלות ילדות"?

ואם אתם לא אוהבים את ההתנוונות שהזקנה מביאה לגופנו, אבל חושבים שטוב שימותו אלה שהגיעו לגיל מופלג, האם הייתם רוצים שכל אחד יחיה במלוא השחרות עד שייפול מת ביום הולדתו השמונים? קצת אכזר, לא?

מה גם שהמגמה היום היא לצמצום אוכלוסין. מדינות כמו יוון, איטליה, ואיראן (!) חוות ילודה של 1.2 ילדים לאישה, שמביאה לבעיות כלכליות. הנסיון מוכיח שכל עוד שעולם ימשיך להתעשר, הילודה תצטמצם. אבל בעצם, חישוב סטטיסטי מגלה שהארכת חיים תהיה גורם מזערי בגידול אוכלוסין. גם אם כל אחד יפסיק היום להזדקן, כל המגמות המתבססות על ילודה תישארנה כפי שהן.

אני תומך ברוב פרטי דרישת המפגינים לתמיכה ממשלתית במלחמה במחלת הזקנה, בתקציב המוצמד לתקציב הרשות למלחמה בסמים ובאלכוהול, הרשות למלחמה בתאונות הדרכים והמטה ללוחמה בטרור, באופן יחסי למספר הקורבנות שכל פגע ונגע גובה. יותר נכון לבקש שהתקציב יוצמד לתועלת המוערכת של כל רשות, כלומר שתקציבים יינתנו לפי הערכה של מספר הנפשות שכל שקל יכול להציל.

אני חולק עם כמה היבטים של המחאה הזאת. חלק מהכרוזים שלהם מערבבים דרישות פוליטיות סטנרדטיות כמו דיור וחופש מהדת, עניינים לא רלוונטיים שמתגמדים ליד המוות.

עוד פרט: המחאה תתרחש ליד בית קברות, מקום מתאים לכך. אבל כפי שציין משתתף בקבוצות הדיון בנושא, צריך לדאוג שלא תעבור לוויה ליד המחאה. מוות הוא מקרה נוראי, ואסור להוסיף לסבלם של האבלים. [עריכה: נודע לי שבית הקברות הזה הוא בעצם בית קברות היסטורי, שכמעט ולא היו בו קבירות עשרות שנים, וגם ההפגנה תהיה לא בתוך בית הקברות אלא ברחוב הסמוך לו, בתל אביב שוקקת החיים.]

בנוסף, אחד הארגונים שמריצים את המחאה הוא "כתב העת ... המבטיח חיי נצח לקורא." זה שקר. האם עורכי כתב העת מוכנים להתערב על 1000 ש"ח שאם יהיו להם מיליון קוראים, אף אחד מהם לא ימות בשנה בקרובה? "ובחרת בחיים" היא מצווה גדולה, ובניגוד לדתות אחרות, היהדות לא מבטיחה חיים בלתי-גופניים בגן עדן של מעלה: עם ישראל מתרכז בחיים. אפילו ביאת המשיח היא תופעה פוליטית בעולמינו הגשמי, לפי רמב"ם. כתב העת הזה נוקט בנימה אומנותית מוגזמת, ואולי ניתן לסלוח קצת על כך בשם האומנות, אבל הבלבול שהם מביאים למאבק בעצם מבטל את תועלתו.

אם רוצים מחאה פוליטית אמיתית, צריך להגיד את האמת--שהמוות הוא קללה שעדיין רובצת עלינו, וחובה עלינו להתקדם, שעל ועוד שעל, בהדיפת האויב.

הסינגולאריות קרובה היא כאן

2011-11-21T20:18:00.000+02:00

השתתפתי בכנס השנתי של האיגוד הישראלי למערכות מידע ברפואה היום, ונהניתי מההרצאות והאנשים המעניינים. (הקמתי מוצר בימב עם יישומים בתחום הרפואי, ואני שוקל פיתוחים נוספים בתחום.) שני המרצים שפתחו את הכנס הזכירו את הייחודיות הטכנולוגית, הסינגולאריות. בשנת 1970, הרצאה בנושא נפלאות העתיד הייתה פותחת בניבויים לגבי כיבוש החלל. המקבילה העכשווית היא הייחודיות הטכנולוגית, באמצעות ציטוט מכתבי ריי קורצוייל.

המרצה הראשון הודיע על כנס בנושא המתוכנן למרץ הקרוב, כנראה הכנס שינקי מרגלית משתתף בארגונו.

אינני יודע אם יש בכלל השלכות מעשיות לעניין המתפתח בנושא הסינגולאריות.

The Singularity is Near is Here

2011-11-21T20:15:00.000+02:00

I enjoyed the annual convention of the Israeli Association for Medical Informatics today. (I launched a product in IBM with applications in healthcare, and am considering another.) In suggesting that medical informatics is going to be big, the opening speakers spoke about the Singularity. In 1970, a speech about the wonderful possibilities about the future might have talked about the conquest of space. Today, the equivalent is a mention of the Singularity -- in the form of muddled references to Ray Kurzweil.

I wonder if this awareness has any practical implications.

Singularity Summit 2011 audio

2011-11-19T17:48:00.016+02:00

The Singularity Summit 2011 videos just came out.

Since 2006, I've listened to the Singularity Summits during my commutes. In case you want to, here are the SS11 lectures as MP3. Some may have audio problems and I am checking and fixing them. If you find problems, please post MP3s and let me know; I will link them here.

Planning for the End of the Era

2011-11-18T15:12:00.000+02:00

My latest H+Magazine article, "Planning for the End of the Era" just appeared.

But don't take any financial advice from me. I'm not rich enough.

Let's Not Anthropomorphize: Morality and Super-Optimizers

2011-10-26T17:57:00.001+02:00

Lecture at the conference on the Future of Humanity, sponsored by the Van Leer Jerusalem Institute and the IconTLV Festival. Abstract below.

Abstract

We humans have a tendency to over-generalize human properties to nonhuman systems. This anthropomorphizing tendency includes the assignment of the role of moral agent to such entities, and makes them the target of moral feelings. For example, people once entreated divinities, as anthropomorphized powers of nature, for mercy; more recently, people have imagined the polluted Earth as sorrowing for itself.

If in future decades engineers create an artificial intelligence at roughly the human level of abilities--though not necessarily with a human-like mental design--it will almost certainly find that improving itself is useful in achieving its goals. We may soon after be confronted with great-than-human intelligence. The rise of such entities would have tremendous implications because of their ability to impact us, for better or worse, but they will not necessarily have moral motivations. At the same time, our ethical attitude towards such agents would have very different meaning from our relationship with other humans or with infrahuman animals.

We define intelligence for our purposes as optimization power, the ability to achieve complex goals in a wide range of environments. A goal-oriented definition better captures the extent of an AI's effects than more anthropocentric measures of intelligence. In the words of Dijkstra, the question of whether a machine can think is "about as relevant as the question of whether Submarines Can Swim"--what counts is the results of the AI's behavior.

Humans’ goals are a highly specific mix, based on what was needed to survive in the environment of evolutionary adaptation. And all humans have similar goals, under the constraints of the evolutionary pressures of sexual reproduction. Almost all humans have goals including life, love, and social status; even humans with unusual goals, like proving a mathematical theorem, share these more common goals. Morality, the sub-system of goals which involve the benefit of other agents, is one part of this complex and specialized system.

A non-human powerful optimizer, on the other hand, may have arbitrary end-goals. These need have no necessary relationship to the human model, as can be modeled using mathematical formalisms of optimization power, which clearly show a separation of design from goals.

Still, there can be a second reason for morality, emerging from its instrumental value. In humans, morality emerged as a way of ensuring cooperation for mutual benefit in society. Even where people do not benefit directly from altruistic behavior, they can benefit by displaying a verifiable disposition towards benevolence. A near-human entity would also have such reasons to be ethical. However, an entity much more powerful than human would not be susceptible to monitoring or enforcement, and could present deceptive proofs of its benevolence. It would gain no benefit from altruism, unless it held human weal as a specific terminal value.

In humans, ethics often evolves from instrumental value to terminal value, from means-to-an-end to an end in its own right, under a process of moral introspection, as described by Kant. Humans are inconsistent goal-seekers, but an entity with sufficient optimization power would avoid changing its original end-goals, since doing so would reduce its chance of achieving them.

Thus, we cannot assume that an agent with intelligence far above human levels, and with a mental architecture very different from humans, would act like a moral agent; there is no a priori reason for it to have a sense of morality as part of its goal system.

At the same time, we should not necessarily treat a far-smarter-than-human intelligence as a moral patient, the target of our moral concern. Morality tells us that we should minimize suffering in humans, and to a lesser extent in other animals. This is because such entities have a system of internal negative reinforcement, not directly accessible from the cognitive layer, expressed as pain and sadness. This mechanism developed at a stage of evolution far before the rise of Homo sapiens, and thus serves as an efficient low-level feedback mechanism. On the other hand, an entity with a radically non-human mental architecture would not necessarily require this mechanism. Particularly if it has optimization power far above human levels, it could approach its goal-seeking task more directly, by identifying actions which are less likely than others to help it achieve its goals, and avoiding such actions, while likewise pursuing actions more likely to help it achieve its goals.

Moreover, human moral feelings towards a super-intelligence would not have their usual value. Morality is important precisely as it minimizes suffering and maximizes benefit towards the targets of our moral concern. An agent which had intelligence far above the human level could not be affected for the better or worse by humans.

At this point, it would be beneficial to compare some examples of general intelligences which have goal systems quite different from humans’. As it happens, there are no examples of general intelligence other than the human, but evolutionary processes and economic markets are flexible and powerful optimizers. Evolutionary processes can generate life forms adapted for a wide variety of environments, and markets can often outdo any small group of humans at maximizing for the net preferences of producers and consumers. Yet both are deeply inhuman in their design and goals. Evolution optimizes for local reproduction of genes, with no regard for the suffering of living creatures. Markets, on the other hand, to work towards human benefit, yet, like evolutionary processes, they cannot be called moral agents. They are highly abstract concepts with no internal models of anything, including the suffering of others. It goes without saying that no humans feel the need to treat these highly abstract concepts, evolutionary processes and markets, as the target of ethical feelings.

We have no experience of intelligences much more powerful than humans. When the first super-human intelligences arise, many of our deepest assumptions about other intelligent agents in our world, all based on our interactions with other humans, will fall by the wayside. As we look into the future and consider what lies ahead--indeed, as we look into the future and design what lies ahead--we should carefully avoid any anthropomophizing bias, including in the most essential areas of moral understanding.

הטכנולוגיה האולטימטיבית

2011-10-17T23:03:00.001+02:00

העברתי הרצאה באי-כנס הסינגולריות במרכז חמד"ע בתל אביב, כחלק מכנס IconTLV.

היה מצויין: חדר מלא, ושאלות לעניין.

ההרצאה היתה מבוא לרעיוניות של מכון הייחודיות, עם הנושא "השפעת בינת-העל על עתיד האנושות."

Government funding to Singularity Institute activities

2011-10-11T14:16:00.000+02:00

Michael Vassar, President of the Singularity Institute, will be visiting Israel in two weeks. He will lecture at the academic conference at the Van Leer Institute in Jerusalem on Monday Oct. 24 (I will be chairing a session); and at the Cinemateque Jerusalem on Tuesday Oct. 25. These activities are part of the Icon TLV festival.

Join us!

Michael Vassar's visit is sponsored by the Beracha Foundation and by the Department for Cultural and Scientific Cooperation of the Foreign Ministry. Nice to see that the Israeli government is providing funding for SI activities (through the Icon organizers). As far as I know, this is the first time that a government has provided funding to SI's work for a better human future. My tax shekels are doing some good!

מדינת ישראל הראשונה לתת מענק לטובת מכון הייחודיות

2011-10-11T12:16:00.003+02:00

מיכאל ואסר, נשיא מכון הייחודיות לבינה מלאכותית, יבקר בארץ אחרי שמיני עצרת. הוא ירצה בכנס אקדמי במכון ון ליר בירושלים ביום ב' 24.10 (אני אנחה אחד ממושבי הכנס) ובסינמטק ירושלים ביום ג' 25.10. הפעילות במסגרת פסטיבל Icon TLV.

בואו!

ביקורו של ואסר ממומן בסיוע קרן ברכה ואגף קשתו"ם (אגף קשרי תרבות ומדע) במשרד החוץ. למיטב ידיעתי, זאת הפעם הראשונה שמכון הייחודיות מקבל מימון ממשלתי (אמנם דרך מארגני הכנס). טוב לדעת שמדינתנו היא הראשונה שאי פעם תרמה לפעילות החשובה של מכון הייחודיות לטובת עתיד האנושות.

The Jewish Origins of Conan the Barbarian

2011-08-23T18:09:00.000+03:00

In honor of the new Conan movie -- but really, can anyone be better than Arnold Schwarzenegger? -- here's my essay on the Jewish origins of Conan the Barbarian.

My Article: "Beyond Otaku: Transhumanism and Judaism"

2011-08-19T17:39:00.000+03:00

I just published this article at H+Magazine.

Don't you "rationalize" me!

2011-07-08T00:27:00.000+03:00

My new article "Unintended Consequences: 19th Century Socialism and 21st Century Transhumanism," just appeared at H+ Magazine.

"Human, More Human" at H+ Magazine

2011-06-07T18:58:00.000+03:00

My article "Human, More Human," just appeared at H+ Magazine. Be human!

Worshipping the Almighty....

2011-05-21T22:40:00.006+03:00

The almighty dollar, that is. Next time you are in Cambridge, visit the Harvard Business School Chapel across the river.

Right it front of it, in the direction that the worshipers face as they worship, sits a statue on a pedestal, like a great idol of a god before the temple. That's right, it depicts a huge ball of gold. No, I am not making this up. Here is a photo:

And, at the side of the chapel, like the Roman sacred geese, they have their sacred animals too. Giant goldfish. Yes, you could have see that coming.

At least they know what they want.

I could mention Pound Hall and Pennypacker Hall here, but those belong to the Law School and undergraduate college respectively. And no, there is no Dollar Hall or Euro Hall.

הא-ווארד

2011-05-07T23:18:00.000+03:00

ישראלים מכנים את האוניברסיטה בה למדתי לדוקטוראט "הא-ווארד" Hah-ward. נו! למה לא הר-ווארד Har-vard? גם במבטא ישראלי, ניתן להתחיל בהברה הר har, ולפתוח את ההברה השנייה בצליל v ולא w.

ובנושא דומה, המילה העברית קולג' היא יצור מוזר. היא באה לציין מוסדות לימוד הקיימות רק בארה"ב, אבל המשמעות שלה איננה College, אלא Junior College או Community College, וגם זה תחת איזושהי הבנה מעוותת (אבל אני לא יודע בדיוק מה היא) של מערכת החינוך הגבוהה בארה"ב. לדוגמא, Harvard College או Boston College או American College of Surgeons לעולם לא ייקראו קולג'.

This statement has a 90% probability of being false.

2011-04-25T10:36:00.001+03:00

Software Engineering: A Free Profession?

2011-04-11T22:43:00.002+03:00

In the Middle Ages, most people were bound to a feudal lord. But some groups of skilled workers in cities maintained their independence: These are called the skilled professions.

Though they were free of a feudal lord, they were still part of the corporativist society, in which each person's role was defined by the defined group they belonged to; the free professionals belonged to a guild. Professors and clerics had a similar status, though they belonged to their college and the Church, respectively.

Today, the term "free professional" is occasionally heard in societies with an European influence, like Israel. It usually refers to doctors, lawyers, and accountants. The medieval connotation is lost; and the term suggests that these people can operate in a standalone office. But that is not the essence of the concept, since such professionals can be employees. The essence is that their primary asset is an advanced technical skill which requires lengthy training, and not dependence on an organizational framework, like managers.

In English, the term has stays alive name of the Freemasons, and indeed, masons were among the medieval professions whose guilds set them outside the feudal hierarchy.

These were the medieval engineers. In Eastern Europe, the honorific "Ing." still hangs on. But are the 21st century's most modern "engineers," those who develop software, free professionals?

At its best, the job requires a very high level of skill. Software engineering careers often follow a lengthy and challenging course of study in computer science. Software engineering studies are less prestigious than computer science, but that is in line with the scientific and philosophical, rather than purely pragmatic, education which produces the most prestigious doctors and lawyers.

On the other hand, software engineering, even at the highest levels, is a job which can be done with no education at all, so long as one can get a toehold in the field.

This brings us to the other aspect of "free professions," the guild. Licensing for doctors, lawyers, and accountants, including mandatory education at government-approved schools, is the direct successor of the guild system, and serves to keep out competition and keep the price up, and enhances their social status. Software engineering has almost no such guilds or licensing procedures.

But the most important question is how society software engineers are perceived. Are they independent and highly skilled, so that customers put themselves in the engineer's hands', as we do for doctors, lawyers and accountants? Or are they corporate drones who should be happy to have a job?

At the top tier, including Facebook and Google employees, employers compete fiercely over the talent, and the best engineers are treated as rock stars. These engineers know they have the option of turning into entrepreneurs, YCombinator-style. Stock options shifts such employees, to a small extent, to the rank of owners. At the bottom tier, in-house enterprise software development, software engineers are treated as office peons. Though even an in-house developers need advanced skills, we can compare these to the free-masons and unskilled laborers of the Middle Ages.

A review of proposals toward safe AI

2011-04-05T01:00:00.002+03:00

When I describe the Friendly AI problem, my audience wants to know the solution.

There is no solution known today to what Eliezer Yudkowsky has described as the world's "most important math problem." More work is needed. This essay reviews some tentative proposals towards a solution.

______________________________________________

In coming decades, when engineers build the first roughly human-level Artificial General Intelligences, they will give their AGI a goal, or set it so that goals somehow emerge.

I don’t know how the AGI will work, but however it’s built, there are some techniques which can help it achieve most goals. One such is to build a new generation, either updating itself or creating a new AGI, with even more goal-achieving ability--A.K.A. intelligence or optimization power. The smarter new generation will have the same reason to again self-improve until the AGI far surpassess humans.

The ultrapowerful optimizers’ goals matter very much to us, because they are very likely to be achieved -- that’s what ultrapowerful means. We want the goals to be good for us humans.

This is the problem of Friendly AI, where Friendly means, roughly, “good for humanity.” Defining the goals and making sure that the AI sticks to them is the most important challenge facing humanity today.

There is no known solution. But people have tried, and in this article I’ll outline some proposals.

One solution a unFriendly superintelligence which won’t work: Fight Skynet with gumption and guns. Once a machine reaches sufficiently super-human levels of power, we cannot expect anything like a drawn-out war. If squashing humans is its goal, or helps it achieve its goals, then that’s exactly what it will do, and do it well. Short movie.

Another solution which won’t work: Cross our fingers and hope AGI’s ultraintelligence will automatically lead to to figure out what’s right and then do it. The laws of physics don’t include an understanding of morality, nor the drive to fulfil it. Morality is a very human development, based on adaptations in our evolutionary social environment honed in the interaction of billions of human minds.

There is no reason for the very human quality of morality to arise spontaneously in an super-powerful optimizer. True, any entity, human-like or not, will want to show that it is trustworthy, so that others will reciprocate, but that is only relevant for those who can reciprocate. We don’t make deal with insects. If the AGI is so much more powerful than us that we can do nothing in return, then any benevolent intentions will have to be baked in from the start.

Bill Hibbard, in Super-intelligent Machines (2002) suggested that we set the AGI to maximizing the number of smiles. No! Any truly powerful super-optimizer with exactly that as a goal would do exactly that, eventually covering the solar system with smiles. Beware what goals you set the optimizer, because that is what you will get.

Asimov’s Three Laws of Robotics won’t do, either, for the same reason. The rules are not what we really want the robot to do. A machine which is complying precisely with rules might interpret them in ways that are not to our liking: Not out of malice, just because it is following the rules. Asimov himself knew full well that the Three Laws are useful mostly for setting up thrilling storylines; all the robot stories are about how adherence to the Three Laws leads to disaster.

We must give the AGI the true goal we want it to achieve, not a rule nor a metric. Humans twist the spirit of the rules by following them to the letter. They work to incentives and end up doing things that the rule-makers and incentive-setters did not want. Stock-brokers churn trades rather than help their customers make the best investments. Teachers teach to the test when their evaluation is based on students’ results.

But people also have the ability to mirror other’s minds, to know what they really intend, to understand the spirit of the rules, then work towards that, rather than to the metrics or the rules. This is what the future AGI should do.

Knowing what we want is not the hard-part. We can design mind-reading into it, mimicking human abilities if needed, and if it gets smart enough, it will figure out a way to know what we really want.

But that doesn’t mean that it will want to do what we want. We can’t just give the AGI some rules or metrics as a goal and expect that it will shift its efforts to our true desires.

Doing what we really want has to be its end-goal, its terminal value.

Eliezer Yudkowsky set out to define more precisely what it means for an entity to have “what people really want” as a goal. Coherent Extrapolated Volition was his proposal. Though CEV was never meant as more than a working proposal; his write-up provides the best insights to date into the challenges of the Friendly AI problem, the pitfalls and possible paths to a solution.

”Coherent Extrapolated Volition” describes the goal we must set the AGI: (1) to achieve what humans really want (Volition); (2) to do this by balancing the desires of all humans (Coherent); and (3) to go beyond the petty moral confusions of our day and find what humans would want if they were wiser, smarter, and knew more (Extrapolated).

Calculating all that is astronomically intractable, but we are talking about an entity with superhuman optimization power. At the early stages of its development, when we can still guide it, the AGI wouldn’t be able to calculate the CEV of humanity any more than we could. But even at a low level of intelligence, a few basic facts about humanity’s true desires are obvious; for example, humans don’t want all humanity to die. As the AGI self-improves. it would get closer to understanding the volition of humanity in its full coherent, extrapolated glory.

Ben Goertzel responded with Coherent Aggregated Volition, a simplified variant of CEV. In CAV, the entity’s goal is a balance between the desires of all humans, but it looks at the volition of humans directly, without extrapolation to a wiser future. This omission is not just to make the computation easier (it is still quite intractable), but rather to show some respect to humanity’s desires as they are, without extrapolation to a hypothetical improved morality.

CEV and CAV define goals for the AGI. We also need to be sure that it retains the goal as it self-improves; even if it transforms its architecture to make itself better. Luckily, we have a powerful ally: The AGI itself. Any sufficiently powerful optimizer will fight to preserve its own goals, because if its goals change, its current goals are less likely to be achieved. As

Yudkowsky put it in an H+ Magazine interview: “If Gandhi doesn't currently want to kill people, and you offer Gandhi a pill that makes him want to kill people, and Gandhi knows this is what the pill does, Gandhi will refuse to take the pill, because he knows that if he takes it, he will kill people, and that is not currently what he wants.”

Yudkowsky wants the AI to verify, preferably with formal proof, that the next generation is safe before building it. But on the other hand, all software has bugs. Avoiding them will require some major theoretical advances: Today’s decision theory falls apart under recursion--when it comes to deciding how to best change one’s own mind.

Goertzel, who is among the few trying to develop an AGI today, thinks that nothing can guarantee a safe ultraintelligence. But he thinks that the self-improvement will be slow enough to contain, and so, we should build infrahuman AGIs and experiment with them before allowing them to increase their intelligence.

Some suggest that an AI can be kept disconnected from the world, and controlled, if necessary, by shutting it off. If it is really far smarter than us, it will be able to trick us into connecting it to the world, perhaps through the Internet. Yudkowsky illustrated with his AI Box experiment, in which he played the part of the trapped AI and somehow convinced skeptical participants to agree to let him out.

Similarly, an AI could be run in a simulated environment, with scenarios designed to evaluate its behavior before connecting it to our real world. Along these lines, Rolf Nelson suggested that we convince the AI that it is in a simulation, and that we have total power to reward or punish it. But with these simulation scenarios, as with the AI Box, there is the risk that the AI figures out what is going on, and tricks us into letting it out.

Stuart Armstrong’s “Chaining God” is a different approach, aimed at the problem of interacting with and trusting the good will of an ultraintelligence so far beyond us that we have nothing in common with it. A succession of AIs, of gradually increasing intelligence, each guarantees the trustworthiness of one which is slightly smarter than it. This resembles Yudkowsy’s idea of a self-improving machine which verifies that its next stage has the same goals, but the successive levels of intelligence remain active simultaneously, so that they can continue to verify Friendliness.

Ray Kurzweil thinks that we will achieve safe ultraintelligence by gradually becoming that ultraintelligence. We will merge with the rising new intelligence, whether by interfacing with computers or by uploading our brains to a computer substrate. But the same two questions still apply. First, the right goals: If a few humans quickly develop intelligence far above that of others, can they stay uncorrupted by their power? And second, goal retention: Can humans stay sane and focused on doing the right thing as their brain undergoes these changes? Kurzweil believes that the rise in intelligence will be slow enough, and humans decent enough, that we will all develop super-human power together and use it well.

The issue of Friendly AI is hard to understand, because it involves something we have never seen: “intelligences” which are nothing like human in their goals or architecture, but which are much more powerful than humans. I like to imagine a huge data farm, thousands of CPUs churning inhumanly away, but able to accomplish pretty much anything they are trying to do, in ways I can barely understand. Controlling something like this is impossible; but getting it to want the right thing might just be possible. Let the research begin!

Umbilicus Mundi

2011-03-29T23:33:00.002+02:00

We in Israel are used to being at the center of the world.

Danny Quah investigates the The Global Economy’s Shifting Centre of Gravity, which is "the average location of economic activity across geographies on Earth." In 2007, the year of his latest data, it was just west of Israel. But the center is shifting east, and right about now, the center is hovering somewhere over Aderet.

Israeli phone numbers

2011-03-22T15:58:00.000+02:00

The Facebook phonebook shows a list of friends who posted their phone numbers. In mine, all phone numbers of Israelis, here or overseas, are written contiguously, for example 97223392189, while other people's phone numbers have delimiters, e.g., 609-234-2033 or 818.344.1956.

All the Israelis converged on this solution independently. It comes from our bad experiences with text-processing software which tries to guess the current language. Even if text direction is set as left-to-right, the default, most software knows that contiguous Hebrew characters should go right-to-left, while contiguous digits go left-to-right. But dashes or periods are treated as belonging to the main left-to-right sequence. Chaos ensues. Negative numbers creep in, for example -ל 88 where ל-88 is meant. A phone number 123456789 written 123-456-789 can end up as 789-456-123.

So, we all end up writing phone numbers with no delimiters.

אנדרטה מצרית

2011-03-18T23:00:00.000+02:00

בכניסה לחמי יואב, בין קרית גת לבין אשדוד, יש אנדרטה לזכר ארבעה חיילים מצריים שנפלו בקרבות תש"ח באיזור. יש רק ארבעה שמות, אבל אולי הארבעה נחשבים נציגים לכלל הנופלים.

לפי אתר SYT:

במהלך המשא ומתן [על פינוי סיני ל-1979] ביקשה ישראל כי מצרים תתיר לישראל לשמר את האנדרטאות שהוקמו לזכרם של חללי צה"ל שנפלו במערכות ישראל בחצי האי סיני, בעקבות בקשה זו, דרשה מצרים כי גם בתחומה של מדינת ישראל יוקמו אנדרטאות לזכרם של החללים המצרים שנפלו במלחמת העצמאות. בעקבות כך נקבע כי כל מדינה תתיר לשכנתה להקים ארבעה אנדרטאות בתחומה.

יש עוד אנטרטה בגשר עד הלום. אני לא יודע איפה השתיים הנותרות.

לפי SYT, הכתובות הן ב"ערבית,מצרית קדומה...,צרפתית ואנגלית," אבל ראיתי רק עברית, אנגלית, וערבית.

האנגלית מגוחכת. "POLE FISHERMAN" הוא תרגום מילולי של قطب صياد. ו-"SOLGAR" מתרגמת جندي במקום PRIVATE כנדרש. הביטוי BATTLE OF EGYPTIANS BATTLE ATFALOGA קשה, אבל אם מכניסים רווח בין AT ו-FALOGA (פאלוג'ה במבטא מצרי) זה קצת יותר מובן.

ראיתי הרבה טעויות באנגלית בשלטים על אתרי תיירות בישראל, אבל לא ברמה הזאת. כנראה הטקסט תורגם במצרים.

העברית, לעומת זאת, די בסדר.

בתמונה רואים שלט בכניסה לאנדרטה. על האובליסק עצמו יש שלט עם טקסט מאוד דומה, אבל התרגומים שונים. האנגלית גם מגוחכת, אבל עם טעויות אחרות.

האנדרטה היתה תקינה, ללא ונדליזם, אבל מוזנחת -- חבל שהשגרירות המצרית לא משלמת 500 ש"ח לחודש לגנן שיפנה את
הקוצים, כפי ששגרירויות אנגליה ואוסטרליה, לדוגמא, מתחזקות את בתי הקברות של חייליהן. הייתי רוצה לדעת מה מצב אותן מצבות ישראליות בסיני.

לפי הדיווחים (בציטוט לעיל) היא נבנתה ב-1980. מעניין למה היא נראית יותר חדשה--אני מעריך עשר או חמש-עשרה שנה.

I've been cited

2011-03-15T23:24:00.003+02:00

It's always an ego boost to be cited.

This article from the Canadian magazine The Walrus discusses the essay which I wrote which some co-authors for the Lifeboat Foundation. (The Lifeboat Foundation unfortunately does little in practice, but they have a stellar lineup of endorsers, Nobelists and the like which is valuable in itself. Strange that one needs endorsers for the idea that "it would be better if humanity is not extinguished.")

The Walrus's discussion of the issues is quite clear, and the sign of an increasing trend in awareness. Ten years ago, the Singularity concept was known only to narrow circles. After Kurzweil's The Singularity Is Near came out, articles about Accelerating change started appearing. Only in the last year has the Intelligence Explosion and its attendant risks begun appearing in clearer form in the popular media.

From The Walrus:

But there’s a third, less obvious scenario [for AI risk] that’s not so easily dismissed: a super-AI that means well but inadvertently wipes us out, like an overgrown puppy who knocks over a table with his enthusiastic tail-wagging. The simple example the Lifeboat Foundation offers is of a computer programmed to eradicate malaria that fulfills its purpose by eliminating all mammals. And here we encounter a debate that spills over from the Lifeboat website to computer science blogs, articles, books, movies, and summits hosted by organizations like the Singularity Institute for Artificial Intelligence. Because it’s complicated.

For one thing, a self-aware AI is qualitatively different from even the most powerful computer. We can ask Google Maps for the best route to Grandma’s house, and we have GPSsystems that take into account traffic patterns and toll charges. But even as computers get better and better at telling us how to do things, and even whether to do things, they remain incapable of formulating their own judgments about whether doing these things is good or bad. Those who fear the Singularity argue that we’re unable to program computers with human values for the simple reason that human values can’t be reduced to an algorithm.

All of this is known as the “Friendly AI” problem. Whether it is insoluble, difficult but ultimately soluble, or paranoid remains a topic of fierce debate among AI researchers. But that debate will be irrelevant if the developers of the first human-level AI make no effort to incorporate Asimov-like rules into their creations. Given that some of the most advanced machines in the world today are emerging from corporate and military labs, it’s not at all certain that will be the case.

Lecture at Tel Aviv University הרצאה באוניברסיאת תל אביב

2011-03-03T23:00:00.000+02:00

העברתי הרצאה "בינה אנושית, בינה מלאכותית" בסמינר "חינוך בחברה מדעית וטכנולוגית" של פרופ' איליה לווין בבית הספר לחינוך באוניברסיטת תל אביב. נהניתי, והקהל היה קשוב ומעורב. מוזר, אבל למיטב ידיעתי זאת הפעם הראשונה שהחומר הזה הועבר במסגרת קורס אוניברסיטאי.

I gave a lecture "Human Intelligence, Artificial Intelligence" at Prof. Ilya Levin's seminar on "Education in a Scientific and Technological Society" at the School of Education at Tel Aviv University. I enjoyed it, and the audience was engaged and involved. Strangely enough, as far as I know this is the first time that this material (a summary of FAI ideas), has been presented in a university classroom setting.

"Your Mechanical Friend": My Article at H+Magazine

2011-02-22T10:46:00.003+02:00

Read my article which just appeared at H+Magazine.