Agents and Environments

• Agent: “anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators” (Russel & Norvig, 2009)
• Human Agent
  • Sensors: eyes, ears, other organs
  • Actuators: hands, legs, vocal tract, others
• Robotic Agent
  • Sensors: cameras, sonar
  • Actuators: motors

Agents and Environments

• Software Agent
  • Input through sensors: keystrokes, file-content, network packages
  • Output through actuators over the environment: displays on screen, write to files, sends network packages
• Percept: Agent’s perceptual inputs at any given instant
• Percept Sequence: History of what the agent has perceived

Agents and Environments

Agents and Environments

• PEAS
  • Performance measure
  • Environment
  • Actuators
  • Sensors

Agents and Environments

• Agent Function
  • Abstract mathematical description
  • Describes the agent’s behavior
  • Maps given percepts sequence to an action
  • \(f: P^* \rightarrow A\)
• Agent Program
  • Concrete implementation
  • Implements the agent’s function
  • Runs on physical architecture

Agents and Environments

• Vacuum-cleaner world example
  • 2 locations
  • Agent percept: which square it is in?
  • Agent percept: dirt in the square?
  • Actions: move left, move right, suck up the dirt, do nothing
  • Example action: if current square is dirty, then suck; otherwise move to the other square
  • Different agents could take different actions according to what they perceive, see figure 2.3
    • What’s the right way to fill the table?
    • What makes a good or bad agent?

Agents and Environments

• Vacuum-cleaner world

Agents and Environments

• Vacuum-cleaner world, partial tabulation of the agent function

Rational Agent

• A Rational Agent is one that does the right thing
• What does it mean - doing the right thing - ?
  • Consider the consequences of the agents behavior
  • Performance measure that evaluates a sequence of environment states
  • What’s a good performance measure?

Rational Agent

• Performance measure for the vacuum-cleaner agent?
  • Amount of dirt cleaned in an amount of time?
    • Maximizes when cleaning, then dumping, then cleaning again, and so on
  • Having clean floor squares at specific times?
  • Penalties for energy consumption and noise generated?
  • Design performance measures according to what we want in the environment, not on how we think the agent should behave
  • Pitfalls
    • Same evaluation measure value for different behaviors
    • Clean floor? - average? - same cleaness all time? - completely clean at times but dirty at other times?

Rationality

• Depends on four things
  • Performance measure that defines criterion of success
  • Agent’s prior knowledge of the environment
  • Actions that the agent can perform
  • Agent’s percept sequence to date

Rational Agent

• Rational Agent - Defined
  • For each possible percept sequence, a rational agent should select an action that is expected to maximize its performance measure, given the evidence provided by the percept sequence and whatever built-in knowledge the agent has.

Is the Vacuum-Cleaner a Rational Agent???

• Performance measure
  • Awards 1 point per clean square at each time step over a lifetime of 1,000 time steps
• Geography of environment: known a priori
• Dirt distribution: unknown
• Initial location of the agent: unknown
• Clean squares stay clean
• Sucking cleans the current square
• Actions: Left, Right, and Suck
• Agent correctly perceives its location
• Agent correctly perceives dirt at a location

Is the Vacuum-Cleaner a Rational Agent???

• Under these circunstances:
• Yes, the agent is rational
• Its performance is at least as high as any other agent’s
• Same agent irrational under other circunstances
  • Once dirt is cleaned it moves from one side to the other (penalty for wasted energy)

Omniscience

• An omniscient agent knows the actual outcome of its actions and can act accordingly
  • Impossible in reality
  • Percepts may not supply all relevant information
• Difference between rationality and omniscience
  • Rationality maximizes expected performance
  • Omniscience maximizes actual performance
• Our definition of rationality does not require omniscience
  • Rational choice depends only on the percept sequence to date

Information Gathering

• Why gather information?
  • Perform actions in order to modify future percepts
  • Important part of rationality
  • Exploration, in unknown environments

Learning

• Rational agent needs more than gathering information
  • Also LEARN as much as possible from what it perceives
  • Starts with prior knowledge or initial configuration
  • Gains experience \(\rightarrow\) modifies configuration, knowledge gets augmented
  • Learns through experience

Autonomy

• If agent only relies on prior knowledge of designer rather than in its own percepts
  • Agent lacks autonomy
• Rational agent should be autonomous
  • Should learn what to do to compensate for partial or incorrect prior knowledge
  • This means learning
  • Vacuum-cleaner: Learn to foresee/predict when dirty comes

Autonomy

• In practice, we don’t provide agents with complete autonomy from start
  • Start with little or no experience
    • Act randomly to get experience
  • Start with initial knowledge + ability to learn
• Ability to learn allows generalization
  • i.e. to succeed even with variations in the environment

Task Environment

• Task envorinments are the problems to which rational agents are the solutions

Specifying the Task Environment

• We specify/describe the Task Environment through PEAS
  • Performance measure
  • Environment
  • Actuators
  • Sensors

Automated Taxi Driver

• PEAS for the Automated Taxi Driver Agent

Other Task Environments

Properties of Task Environments

• Fully Observable vs Partially Observable vs Unobservable
  • Sensors get complete state of environment at each point in time?
  • All aspects relevant to the choice of action
  • Relevance depends on performance

Properties of Task Environments

• Fully Observable Environment
  • No need to keep internal state to keep track of world
• Partially Observable Environment
  • Noisy environment or inaccurate sensors
  • Sensors don’t get all of the environment’s data
• Unobservable Environment
  • The agent has no sensors at all

Properties of Task Environments

• Single Agent vs Multi-Agent Environments
  • Chess game
    • A two-agent environment
  • Cross word puzzle agent
    • A single-agent environment
  • How the automated taxi agent sees other vehicles?
    • As agents?
    • As objects?

Properties of Task Environments

• Competitive Environment
  • Like in Chess: Agent A tries to maximize its performance measure, which minimizes Agent B’s performance measure
• Cooperative Environment
  • Like in the taxi-driving environment \(\rightarrow\) partially cooperative
  • Avoiding collisions maximizes all agent’s performance measures
  • Communication required in multi-agent environments

Properties of Task Environments

• Deterministic vs Stochastic Environment
  • Deterministic if next state completely determined by current state and action taken
  • Otherwise, stochastic
• Fully observable deterministic environment
  • Agent does not worry about uncertainty
• Partially observable environment
  • May appear to be stochastic, in practice they are treated as stochastic (i.e. taxi-driving)

Properties of Task Environments

• Uncertain Environment
  • When the environment is not fully observable or not deterministic
• Nondeterministic Environment
  • One in which actions are characterized by their possible outcomes
  • No probabilities are attached to those outcomes

Properties of Task Environments

• Episodic vs Sequential Environment
  • Episodic: Agent’s experience divided into atomic episodes
    • At each episode: Agent receives percepts and performs a single action
    • Next episode doesn’t depend on actions taken in previous episodes
    • Many classification tasks are episodic
    • i.e. Agent that identifies defective parts on an assembly

Properties of Task Environments

• Episodic vs Sequential Environment
  • Sequential: Current decision could affect future decisions
    • i.e. Chess and taxi-driving are sequential
    • Short term actions may have long-term consequences
  • Episodic are simpler than sequential
    • Agents don’t have to think ahead

Properties of Task Environments

• Static vs Dynamic Environment
  • Dynamic: The environment may change while the agent is deliberating (taxi-driving)
  • Static: Environment doesn’t change while the agent is deliberating (crossword pruzzles)
    • Easy: don’t keep looking at the world while deciding which action it will take
  • Semidynamic: Environment doesn’t change but agent performance’s score does change (chess playing with clock)

Properties of Task Environments

• Discrete vs Continuous
  • Applies to: state of the environment, the way time is handled, and to percepts and actions of the agent
  • Chess is discrete
    • Finite number of states (not considering the clock)
    • Finite/discrete number of percepts and actions
  • Taxi-driving is continuous
    • Speed and location take continuous values
    • Taxi actions are also continuous: steering angles

Properties of Task Environments

• Known vs Unknown
  • Refers to the Agent’s state of knowledge of laws of physics of the environment
  • In known environment outcomes of actions are given
  • In unknown environment agent learns the outcomes of actions \(\rightarrow\) to make correct decisions

Properties of Task Environments

• Hardest environment?

Properties of Task Environments

• Hardest environment?
  • Partially observable
  • Multi-agent
  • Stochastic
  • Sequential
  • Dynamic
  • Continuous
  • Unknown

Properties of Task Environments

Properties of Task Environments

• Environment Class
  • Different environments for the same task
  • Taxi-driving with different traffic, roads, lighting, weather conditions, etc.

Agent Program

• The Agent Program implements the Agent Function
• Mapping from Percepts to Actions

• Architecture: The program runs on a Computing Device with physical sensors and actuators

• \(Agent = Architecture + Program\)
• Agent Program
  • Input: current percept
  • Output: send action to actuators

Agent Program

• Design a rational agent
  • Construct table with appropriate action for every possible percept

Agent Program

• Table agent
  • \(P\) is the set of possible percepts
  • \(T\) is the lifetime of the agent (number of percepts it receives)
  • Lookup table will contain \(\sum_{t=1}^{T} |P|^{t}\) entries
    • For taxi-driving agent
    • Will grow a lot!
    • 1 camera: 30 frames per second, about \(10^{250,000,000,000}\) of entries for 1 hour of driving

Simple Reflex Agent

• Strategy
  • Represent the most important
  • With condition-action rules
  • If car-in-front-is-breaking then initiate breaking
• They are simple but with limited intelligence
  • Problems when environment is not fully observable

Simple Reflex Agent

Simple Reflex Agent

Simple Reflex Agent

Model-based Reflex Agent

• Keep an internal state
  • Keep track of part of the world (what it has seen but can’t see now)
  • Uses the percept history
  • Reacts to unobserved aspects on current state
• Keeps a model of the world
  • Model-based Agent
• The model should keep information about
  • How the world evolves
  • How the agent’s actions affect the world

Model-based Reflex Agent

Model-based Reflex Agent

Goal-based Agent

• Not enough knowing the current state of the environment to decide what to do
  • Agent requires some goal information
  • Goal describes desirable situations
• Finding sequences of actions varies in complexity
  • Search and planning in AI
• Considers the future, not like condition-action rules
  • What would happen if I take action X?
  • Will that make me happy?
• Agents flexible with respect to the environment
  • Modifies its behavior based on received stimuli

Goal-based Agent

Utility-based Agent

• Goals are not enough to generate a high quality behavior
  • Optimal sequences needed to reach the goal
• Utility Function
  • Function \((f_{U})\) that relates a state to a numerical real value to characterize the satisfaction of the agent
  • An internalization of the performance measure

Utility-based Agent

• Expected Utility
  • Agent chooses action that maximizes expected utility of the actions outcomes
  • The utility that the agent expects to generate (on average) given the probabilities and utilities of each outcome

Utility-based Agent

• Correct specification of utility function allows taking rational decisions in 2 cases
  • Achieving some goals implies a conflict, only one of them can be obtained, tradeoff
  • When there are several goals, which one should the agent choose?

Utility-based Agent

Learning Agent

• Learning agent has four conceptual components
  • Learning element
    • Responsible for making improvements
  • Performance element (which selects the actions to take)
  • Critic (how the agent is doing)
  • Problem generator
    • Suggests actions that will lead to new and informative experiences
    • Allows exploration
    • Suggests exploratory actions
  • Responsible for selecting external actions

Learning Agent

How the Components of Agent Programs Work

• Representations
  • Atomic
  • Factored
  • Structured

How the Components of Agent Programs Work

How the Components of Agent Programs Work

• Atomic representation
  • Each state of the world is indivisible, with no internal structure
  • Reduce one state of the world to to the name of a city (finding-driving-route)
• Algorithms for search and game-playing, Hidden Markov models, and Markov decision processes work with atomic representations

How the Components of Agent Programs Work

• Factored representation
  • Splits up each state into a fixed set of variables or attributes, each with a value
  • We can represent uncertainty (i.e. leave an attribute without value)
• Algorithms for constraint satisfaction, propositional logic, planning, Bayesian networks, machine learning use factored representations

How the Components of Agent Programs Work

• Structured representation
  • We can describe relations between objects
• Used in relational databases and first-order logic, first-order probability models, knowledge-based learning, natural language understanding

How the Components of Agent Programs Work

• Structured representations are more expressive than factored representations, which are more expressive than atomic representations

Assignment