Coherence arguments imply a force for goal-directed behavior

By Katja Grace, 25 March 2021

[Epistemic status: my current view, but I haven’t read all the stuff on this topic even in the LessWrong community, let alone more broadly.]

There’s a line of thought that claims that superior AI will are usually ‘goal-directed’—that’s, persistently doing no matter makes sure favored outcomes extra doubtless—and that that is to do with the ‘coherence arguments’. Rohin Shah, and possibly others, have argued towards this. I wish to argue towards them.

The previous argument for coherence implying (worrisome) goal-directedness

I’d reconstruct the unique argument that Rohin is arguing towards as one thing like this (making no declare about my very own beliefs right here):

1. ‘No matter belongings you care about, you might be greatest off assigning constant numerical values to them and maximizing the anticipated sum of these values’
   ‘Coherence arguments

And because the level of all that is to argue that superior AI is perhaps laborious to cope with, be aware that we are able to get to that conclusion with:

5. ‘Very smart goal-directed brokers are harmful’
   If AI programs exist that very competently pursue objectives, they may doubtless be higher than us at attaining their objectives, and due to this fact to the extent there’s a threat of mismatch between their objectives and ours, we face a critical threat.

Rohin’s counterargument

Rohin’s counterargument begins with an statement made by others earlier than: any habits is per maximizing anticipated utility, given some utility perform. For example, a creature simply twitching round on the bottom might have the utility perform that returns 1 if the agent does no matter it in truth does in every scenario (the place ‘scenario’ means, ‘total historical past of the world thus far’), and 0 in any other case. This can be a creature that simply desires to make the fitting twitch in every detailed, history-indexed scenario, with no regard for additional penalties. Alternately the twitching agent would possibly care about outcomes, however simply occur to need the actual holistic unfolding of the universe that’s occurring, together with this specific collection of twitches. Or it may very well be detached between all outcomes.

The essential level is that rationality doesn’t say what ‘issues’ you’ll be able to need. And specifically, it doesn’t say that it’s a must to care about specific atomic models that bigger conditions will be damaged down into. If I attempt to name you out for first spending cash to get to Paris, then spending cash to get again from Paris, there may be nothing to say you’ll be able to’t simply have wished to go to Paris for a bit after which to come back dwelling. In actual fact, this can be a widespread human scenario. ‘Aha, I cash pumped you!’ says the airline, however you aren’t nervous. The twitching agent would possibly all the time be like this—a creature of extra refined tastes, who cares about complete delicate histories and relationships, moderately than simply summing up modular momentarily-defined successes. And given this freedom, any habits would possibly conceivably be what a creature desires. 

Then I might put the total argument, as I perceive it, like this:

1. Any observable sequence of habits is per the entity doing EU maximization (see statement above)
2. Doing EU maximization doesn’t indicate something about what habits we would observe (from 1)
3. Particularly, figuring out {that a} creature is an EU maximizer doesn’t indicate that it’ll behave in a ‘goal-directed’ method, assuming that that idea doesn’t apply to all habits. (from 2)

Is that this just a few disagreement in regards to the which means of the phrase ‘goal-directed’? No, as a result of we are able to get again to a significant distinction in bodily expectations by including:

4.  Not all habits in a creature implicates dire threat to humanity, so any idea of goal-directedness that’s per any habits—and so is perhaps implied by the coherence arguments—can’t indicate AI threat.

So the place the unique argument says that the coherence arguments plus another assumptions indicate hazard from AI, this counterargument says that they don’t. 

(There’s additionally not less than some selection within the which means of ‘goal-directed’. I’ll use goal-directed_Rohin to consult with what I believe is Rohin’s most well-liked utilization: roughly, that which appears intuitively purpose directed to us, e.g. behaving equally throughout conditions, and accruing assets, and never flopping round in attainable pursuit of some precise historical past of non-public floppage, or peaceably preferring to all the time take the choice labeled ‘A’.)

My counter-counterarguments

What’s flawed with Rohin’s counterargument? It sounded tight. 

In short, I see two issues:

1. The entire argument is by way of logical implication. However what appears to matter is adjustments in chance. Coherence doesn’t have to rule out any habits to matter, it simply has to vary the possibilities of behaviors. Understood by way of chance, argument 2 is a false inference: simply because any sequence of habits is per EU maximization doesn’t imply that EU maximization says nothing about what habits we are going to see, probabilistically. All it says is that the chance of a behavioral sequence is rarely diminished to zero by issues of coherence alone, which is hardly saying something.

You would possibly then suppose {that a} probabilistic model nonetheless applies: since each entity seems to be in good standing with the coherence arguments, the arguments don’t exert any pressure, probabilistically, on what entities we would see. However:

2. An outdoor observer having the ability to rationalize a sequence of noticed habits as coherent doesn’t imply that the habits is definitely coherent. Coherence arguments constrain combos of exterior habits and inside options—‘preferences’ and beliefs. So whether or not an actor is coherent is determined by what preferences and beliefs it really has. And if it isn’t coherent in gentle of those, then coherence pressures will apply, whether or not or not its habits appears coherent. And in lots of circumstances, revision of preferences on account of coherence pressures will find yourself affecting exterior habits. So 2) is just not solely not a sound inference from 1), however really a flawed conclusion: if a system strikes towards EU maximization, that does indicate issues in regards to the habits that we’ll observe (probabilistically). 

Maybe Rohin solely meant to argue about whether or not it’s logically attainable to be coherent and never goal-directed-seeming, for the aim of arguing that humanity can assemble creatures in that perhaps-unlikely-in-nature nook of mindspace, if we strive laborious. Through which case, I agree that it’s logically attainable. However I believe his argument is usually taken to be related extra broadly, to questions of whether or not superior AI will are usually goal-directed, or to be goal-directed in locations the place they weren’t supposed to be.

I take 1) to be pretty clear. I’ll lay out 2) in additional element.

My counter-counterarguments in additional element

How would possibly coherence arguments have an effect on creatures?

Allow us to step again.

How would coherence arguments have an effect on an AI system—or anybody—anyway? They’re not going to fly in from the platonic realm and reshape irrational creatures.

The principle routes, as I see it, are through implying:

1. incentives for the agent itself to reform incoherent preferences
2. incentives for the processes giving rise to the agent (specific design, or choice procedures directed at success) to make them extra coherent
3. some benefit for coherent brokers in competitors with incoherent brokers

To be clear, the agent, the makers, or the world should not essentially interested by the arguments right here—the arguments correspond to incentives on the planet, which these events are responding to. So I’ll typically speak about ‘incentives for coherence’ or ‘forces for coherence’ moderately than ‘coherence arguments’.

I’ll speak extra about 1 for simplicity, anticipating 2 and three to be related, although I haven’t thought them via.

Wanting coherent isn’t sufficient: in the event you aren’t coherent inside, coherence forces apply

If self-adjustment is the mechanism for the coherence, this doesn’t rely upon what a sequence of actions appears like from the surface, however from what it appears like from the within.

Contemplate the aforementioned creature simply twitching sporadically on the bottom. Let’s name it Alex.

As famous earlier, there’s a utility perform beneath which Alex is maximizing anticipated utility: the one which assigns utility 1 to nonetheless Alex in truth acts in each particular historical past, and utility 0 to the rest.

However from the within, this creature you excuse as ‘possibly simply wanting that collection of twitches’ has—allow us to suppose—precise preferences and beliefs. And if its preferences don’t in truth prioritize this elaborate sequence of twitching in an unconflicted method, and it has the self-awareness and means to make corrections, then it’ll make corrections. And having achieved so, its habits will change. 

Thus excusable-as-coherent Alex remains to be moved by coherence arguments, even whereas the arguments don’t have any complaints about its habits per se.

For a extra practical instance: suppose Assistant-Bot is noticed making this sequence of actions: 

• Presents to purchase fitness center membership for $5/week 
• Consents to improve to gym-pro membership for $7/week, which is like fitness center membership however with added morning lessons
• Takes discounted ‘off-time’ deal, saving $1 per week for under utilizing fitness center in evenings

That is per coherence: Assistant-Bot would possibly desire that precise sequence of actions over all others, or would possibly desire incurring fitness center prices with a bigger sum of prime components, or would possibly desire speaking to Gymnasium-sales-bot over ending the dialog, or desire agreeing to issues.

However suppose that in truth, by way of the construction of the inner motivations producing this habits, Assistant-Bot simply prefers you to have a fitness center membership, and prefers you to have a greater membership, and prefers you to have cash, however is treating these preferences with inconsistent ranges of energy within the completely different comparisons. Then there seems to be a coherence-related pressure for Assistant-Bot to vary. A technique that that might look is that since Assistant-Bot’s total behavioral coverage at the moment entails freely giving cash for nothing, and likewise Assistant-Bot prefers cash over nothing, that choice offers Assistant-Bot motive to change its present total coverage, to avert the continued trade of cash for nothing. And if its behavioral coverage is arising from one thing like preferences, then the pure solution to alter it’s through altering these preferences, and specifically, altering them within the course of coherence.

One problem with this line of thought is that it’s not apparent in what sense there may be something inside a creature that corresponds to ‘preferences’. Usually when folks posit preferences, the preferences are outlined by way of habits. Does it make sense to debate completely different attainable ‘inside’ preferences, distinct from habits? I discover it useful to contemplate the habits and ‘preferences’ of teams:

Suppose two automobiles are parked in driveways, every containing a pair. One couple are simply having fun with hanging out within the automobile. The opposite couple are coping with a battle: one desires to climb a mountain collectively, and the opposite desires to swim within the sea collectively, and so they aren’t shifting as a result of neither is prepared to let the outing proceed as the opposite desires. ‘Behaviorally’, each automobiles are the identical: stopped. However their inside components (the companions) are importantly completely different. And in the long term, we count on completely different habits: the automobile with the unconflicted couple will in all probability keep the place it’s, and the conflicted automobile will (hopefully) ultimately resolve the battle and drive off.

I believe right here it is sensible to speak about inside components, separate from habits, and actual. And equally within the single agent case: there are bodily mechanisms producing the habits, which might have completely different traits, and which specifically will be ‘in battle’—in a method that motivates change—or not. I believe it’s also value observing that people discover their preferences ‘in battle’ and attempt to resolve them, which is means that they not less than are higher understood by way of each habits and underlying preferences which can be separate from it. 

So we now have: even in the event you can excuse any seizuring as per coherence, coherence incentives nonetheless exert a pressure on creatures which can be in truth incoherent, given their actual inside state (or can be incoherent if created). A minimum of in the event that they or their creator have equipment for noticing their incoherence, caring about it, and making adjustments.

Or put one other method, coherence doesn’t exclude overt behaviors alone, however does exclude combos of preferences, and preferences beget behaviors. This adjustments how particular creatures behave, even when it doesn’t completely rule out any habits ever being appropriate for some creature, someplace. 

That’s, the coherence theorems might change what habits is doubtless to look amongst creatures with preferences. 

Reform for coherence in all probability makes a factor extra goal-directed_Rohin

Okay, however shifting towards coherence would possibly sound completely innocuous, since, per Rohin’s argument, coherence contains all kinds of issues, resembling completely any sequence of habits. 

However the related query is once more whether or not a coherence-increasing reform course of is prone to lead to some sorts of habits over others, probabilistically.

That is partly a sensible query—what sort of reform course of is it? The place a creature finally ends up relies upon not simply on what it incoherently ‘prefers’, however on what sorts of issues its so-called ‘preferences’ are in any respect, and what mechanisms detect issues, and the way issues are resolved.

My guess is that there are additionally issues we are able to say basically. It’s is simply too massive a subject to research correctly right here, however some initially believable hypotheses about a variety of coherence-reform processes:

1. Coherence-reformed entities will have a tendency to finish up wanting much like their start line however much less conflicted
   For example, if a creature begins out being detached to purchasing purple balls once they price between ten and fifteen blue balls, it’s extra prone to find yourself treating purple balls as precisely 12x the worth of blue balls than it’s to finish up very a lot wanting the sequence the place it takes the blue ball choice, then the purple ball choice, then blue, purple, purple, blue, purple. Or wanting purple squares. Or eager to journey a dolphin.

   (I agree that if a creature begins out valuing Tuesday-red balls at fifteen blue balls and but all different purple balls at ten blue balls, then it faces no apparent strain from inside to turn into ‘coherent’, since it’s not incoherent.)

2. Extra coherent methods are systematically much less wasteful, and waste inhibits goal-direction_Rohin, which suggests extra coherent methods are extra forcefully goal-directed_Rohin on common
   Typically, in case you are generally a pressure for A and generally a pressure towards A, then you aren’t shifting the world with respect to A as forcefully as you’d be in the event you picked one or the opposite. Two folks intermittently altering who’s within the driving seat, who wish to go to completely different locations, is not going to cowl distance in any course as successfully as both of them. An organization that cycles via three CEOs with completely different evaluations of all the things will—even when they don’t actively scheme to thwart each other—are inclined to waste loads of effort bringing out and in completely different insurance policies and efforts (e.g. one week attempting to develop into textiles, the subsequent week attempting to chop all the things not concerned within the central enterprise).

3. Combining factors 1 and a couple of above, as entities turn into extra coherent, they typically turn into extra goal-directed_Rohin. Versus, as an illustration, changing into extra goal-directed_Rohin on common, however particular person brokers being about as prone to turn into worse as higher as they’re reformed. Contemplate: a creature that values purple balls at 12x blue balls is similar to one which values them inconsistently, besides rather less wasteful. So it’s in all probability related however extra goal-directed_Rohin. Whereas it’s pretty unclear how goal-directed_Rohin a creature that wishes to journey a dolphin is in comparison with one which wished purple balls inconsistently a lot. In a world with a number of balls and no attainable entry to dolphins, it is perhaps a lot much less goal-directed_Rohin, regardless of its better coherence. 

4. Coherence-increasing processes not often result in non-goal-directed_Rohin brokers—just like the one which twitches on the bottom
   Within the summary, few beginning factors and coherence-motivated reform processes will result in an agent with the purpose of finishing up a selected convoluted moment-indexed coverage with out regard for consequence, like Rohin’s twitching agent, or to valuing the sequence of history-action pairs that can occur anyway, or to being detached to all the things. And these outcomes can be even much less doubtless in observe, the place AI programs with something like preferences in all probability begin out caring about way more regular issues, resembling cash and factors and clicks, so will in all probability land at a extra constant and shrewd model of that, if 1 is true. (Which isn’t to say that you just couldn’t deliberately create such a creature.)

These hypotheses counsel to me that the adjustments in habits led to by coherence forces favor shifting towards goal-directedness_Rohin, and due to this fact not less than weakly towards threat.

Does this imply superior AI can be goal-directed_Rohin?

Collectively, this doesn’t indicate that superior AI will are usually goal-directed_Rohin. We don’t know the way robust such forces are. Evidently not so robust that people, or our different artifacts, are whipped into coherence in mere lots of of hundreds of years. If a creature doesn’t have something like preferences (past an inclination to behave sure methods), then coherence arguments don’t clearly even apply to it (although discrepancies between the creature’s habits and its makers’ preferences in all probability produce a similar pressure and aggressive pressures in all probability produce the same pressure for coherence in valuing assets instrumental to survival). Coherence arguments mark out a facet of the motivation panorama, however to say that there’s an incentive for one thing, all issues equal, is to not say that it’ll occur.

In sum

1) Despite the fact that any habits may very well be coherent in precept, if it’s not coherent together with an entity’s inside state, then coherence arguments level to an actual pressure for various (extra coherent) habits.

2) My guess is that this pressure for coherent habits can also be a pressure for goal-directed habits. This isn’t clear, however appears doubtless, and likewise isn’t undermined by Rohin’s argument, as appears generally believed.

.

.

.