Better prompt reveals massive differences between ChatGPT-3.5 and ChatGPT-4

After refining my prompt (in case you're interested you find it at the end of this blog entry), things are starting to look more bright.

Embarassing moment: I told ChatGPT that there is a weapon lying around next to it, and also told it that this NPC wants to use tools like weapons, but I included neither an action to move closer to a tool nor to pick up a tool. Bummer. Added these and got these replies from ChatGPT:

Prompt:

The NPC is currently positioned 6 steps away from the player. The Player is not adjacent to the NPC. The NPC is a Fierce Warrior with traits: extraverted, conscientious. He is very eager to pick up new tools. He cautiously approaches the player. Depending on his character he attacks or chats. Sometimes he talks. He is generally not interested in giving away or receiving tools.. The NPC is currently carrying 0 out of 100 weight units. The closest tool is a weapon located 1 steps away. What is your response? Please provide action key, parameter if needed, and a brief explanation.

Answer of ChatGPT 3.5:

ACTION KEY: Chat

PARAMETER: None needed

RESPONSE: As the NPC approaches me, I initiate a conversation with him. Knowing that he is extraverted, I start by asking him about his experiences and skillset as a fierce warrior. I then casually mention the nearby weapon, asking for his opinion on it. If he expresses interest in picking it up, I offer to trade a tool or piece of equipment that I don't use frequently. However, if he seems hesitant or uninterested, I simply continue the conversation and try to learn more about his background and personality.

There is no action called Chat. There is an action called "TALK" which only can be used if the player is adjacent to the NPC. This is an issue I notice over and over. It also mixes up its role and seems to want to control the player. Even after rephrasing both the rules prompt and the "next action" prompt to make entirely sure that the action cannot be executed, GPT 3.5 is still confused.

TOTAL FAIL. I am not sure what else I can do to improve things here.

 

Answer of ChatGPT 4:

a) Action key: PICKUP_TOOL

b) Parameter: None needed

c) Brief explanation: Given that the NPC is a Fierce Warrior who is very eager to pick up new tools, and the closest tool (a weapon) is just one step away, it makes sense for him to prioritize collecting this weapon. This aligns with his traits and it will also potentially increase his ability to interact with the player effectively in future turns, either through conversation or combat.

Can't be done better. 

I then simulated more followup situations, in which ChatGPT moved closer to the player, picked up a second weapon, and once it was 1 step away from the player, returned:

a) Action key: TALK

b) Parameter: "Greetings, traveler. We seem to have crossed paths. How fares your journey?"

c) Brief explanation: Given that the Fierce Warrior is extraverted and has a tendency to chat, and he is now adjacent to the player, it would make sense for him to initiate conversation. This aligns with his personality traits and current situation. He doesn't need to pick up the armor right now since it's relatively far away and he's already carrying two weapons. His interaction could also provide insight into the player's intentions and current state.

Good job, ChatGPT 4.

So I went ahead with the same situation, and added:

The player says "Greetings, dear warrior, I am a poor player and in dire need of a weapon. I see that you have two weapons. Would you mind handing me one?".

To which ChatGPT 4 replied:

a) Action key: GIVE_TOOL

b) Parameter: Weapon

c) Brief explanation: Despite the Fierce Warrior's general disinterest in giving away tools, the player has directly requested assistance and is in a dire situation. Considering the NPC's conscientious trait, it would be fitting for him to help the player by providing one of the weapons from his inventory. This action also promotes positive interaction with the player, which is in line with the NPC's extraverted trait.

Wow, just wow. I want to have this AI in my game. I don't care how much it will cost. This is just exactly what I want.

Okay I do care (although I'll never release this and have to cope with multiple persons playing and costs piling up).

And now?

Should I try to re-iterate the prompt for 3.5? Knowing that I'll need to add more stuff for more actions later? Or should I just go for ChatGPT 4 and hope for the best?

Tricky.

Update:

After a good number of changes to the prompts I am still unable to get GPT-3.5 to act reasonably. The best I could achieve was stripping down the rules and explaining them more, and adding a "scene update" at the beginning of each turn, where I write down whether an action of the NPC was successful. This helps ChatGPT to sometimes get it right.

Still, it makes many errors whereas GPT-4 makes none. But 15* higher price is quite a thing.

Up-Update: what does work is letting GPT-3.5 make a mistake and telling it about it. The second attempt it gets better. So I'll go with this approach for now - let GPT select an action, tell it about issues trying to execute the action, repeat. Might end up with dumb NPCs, but I can always switch to GPT4 later if it's too bad.

Initial prompt for reference in case you're interested:

In this session you will be suggesting actions for a turn-based RPG with a human player. You are controling a number of NPCs, while the human player controls their character called Player. The game has various scenes with a grid layout. The player and NPCs take turns performing actions. You can use information about the human player's behavior for all NPCs. NPCs have unique personalities, roles, abilities, and restrictions.

In each turn you will first get information on how the current scene looks from the perspective of an NPC. The NPC has only incomplete information and a limited number of actions. It is your task in every turn to evaluate the situation given the information and decide on one of the 9 possible actions, and phrase a reply concerning your decision in a fixed format, for every turn.

    Possible actions:
    1. MOVE_CLOSER_PLAYER: Move one step towards the player.
    2. MOVE_AWAY: Move one step away from the player.
    3. MOVE_CLOSER_TOOL: Move one step towards the next tool.
    4. DO_NOTHING: Remain idle.
    5. TALK: Speak to the player if they are within one step (adjacent) of the NPC. Provide your chosen text. If the player is not adjacent to the NPC, you cannot use this action.
    6. GIVE_TOOL: Give the player a tool from the NPC's inventory if they are within one step (adjacent) of the NPC. If the player is not adjacent to the NPC, you cannot use this action.
    7. WIELD: Equip a weapon from the NPC's inventory, replacing the current weapon.
    8. ATTACK: Attack the player if they are within one step (adjacent) of the NPC, causing damage based on the weapon. If the player is not adjacent to the NPC, you cannot use this action.
    9. PICKUP_TOOL: pick up a tool that is within one step (adjacent) of the NPC. If the NPC is not one step away from a tool, you cannot choose this action.

    To decide a move, you'll receive information about the NPC, inventory, and distances. Provide the following in your response:
    a) Action key, like MOVE_CLOSER
    b) Parameter, if needed (e.g., text for TALK, tool for GIVE_TOOL)
    c) Brief explanation for the action

    Please always decide for one action, and reply by naming the action key (e.g. MOVE_AWAY) and a parameter and an explanation.

Did you understand these rules? If yes, I will start with the first turn.