% Upper-case    A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
% Lower-case    a b c d e f g h i j k l m n o p q r s t u v w x y z
% Digits        0 1 2 3 4 5 6 7 8 9
% Exclamation   !           Double quote "          Hash (number) #
% Dollar        $           Percent      %          Ampersand     &
% Acute accent  '           Left paren   (          Right paren   )
% Asterisk      *           Plus         +          Comma         ,
% Minus         -           Point        .          Solidus       /
% Colon         :           Semicolon    ;          Less than     <
% Equals        =           Greater than >          Question mark ?
% At            @           Left bracket [          Backslash     \
% Right bracket ]           Circumflex   ^          Underscore    _
% Grave accent  `           Left brace   {          Vertical bar  |
% Right brace   }           Tilde        ~

% ---------------------------------------------------------------------|
% --------------------------- 72 characters ---------------------------|
% ---------------------------------------------------------------------|
%
% Optimal Foraging Theory Revisited: Chapter 3. Optimization Objectives
%
% (c) Copyright 2007 by Theodore P. Pavlic
%
 
\chapter{Statistical Optimization Objectives for Solitary Behavior}
\label{ch:optimization_objectives}

The efficacy of any particular behavior may be measured quantitatively
in various ways. In this chapter, we approach the problem of combining
appropriate statistics so that the utility of solitary behaviors can be
measured for a given application. Choosing a static behavior to maximize
some unit of expected value is analogous to choosing investments to
maximize future returns. Reflecting this analogy, behavioral ecology has
borrowed methods from investment theory and capital budgeting for
behavioral analysis. We also use these methods, collectively known as
\index{modern portfolio theory (MPT)|see{finance, modern portfolio
theory}}\index{MPT|see{finance, modern portfolio
theory}}\acro[\index{finance!modern portfolio theory
(MPT)|indexdef}modern portfolio
theory~(MPT)]{MPT}{\index{finance"!modern portfolio theory
(MPT)"|indexglo}modern portfolio theory}, to analyze our model; however,
we generalize the classical \ac{OFT} approach. This approach not only
allows it to be applied to engineering problems, but it also provides
answers to some of the criticisms of the theory.  Additionally, we
suggest new ways of describing optimal agent behavior and relationships
among existing methods.

The major purpose of this chapter is to introduce functions that combine
statistics of the agent model to measure the utility of solitary
behaviors. Behaviors that maximize these functions may be called
\emph{optimal}. In \longref{sec:obj_fn_structure}, we define the
structure of the optimization functions that are interesting to us.  In
\longref{sec:oft_approach_optimization}, we describe the optimization
approach used frequently in classical \ac{OFT}. In
\longref{sec:alternate_optimization_objectives}, we propose an alternate
approach and give new or refined optimization objectives for analyzing
agent behavior. Finally, in \longref{sec:future_econ_directions}, we
briefly discuss how insights from \index{post-modern portfolio theory
(PMPT)|see{finance, post-modern portfolio
theory}}\index{PMPT|see{finance, post-modern portfolio
theory}}\acro[\index{finance!post-modern portfolio theory
(PMPT)|indexdef}post-modern portfolio
theory~(PMPT)]{PMPT}{\index{finance"!post-modern portfolio theory
(PMPT)"|indexglo}post-modern portfolio theory} may inspire new
optimization approaches in both agent design in engineering and agent
analysis in biology. All results discussed in this chapter will be
qualitative and justified graphically. Specific analytical optimization
results for some of the objectives discussed here are given in
\longref{ch:optimization_results}.

\section{Objective Function Structure}
\label{sec:obj_fn_structure}

Optimization functions usually combine multiple optimization
\emph{objectives} in a way that captures the relative value of each of
those objectives. In our case, each of our objectives is a statistic
taken from the model in \longref{ch:model}. Therefore, in
\longref{sec:stats_of_interest}, we present statistics that could serve
as objectives for optimization and methods for combining them. In
\longref{sec:optimization_constraints}, we discuss motivations for
constraining the set of feasible behaviors and show how these
constrained sets can be incorporated into optimization.  Finally, in
\longref{sec:optimization_choice_effects}, we discuss the importance of
exploring a variety of optimization criteria.

\subsection{Statistics of Interest}
\label{sec:stats_of_interest}

\longref{tab:common_stats} shows some obvious choices for statistics to
be used as optimization objectives.
%
\begin{table}\centering
\begin{tabular}{@{}r*{4}{c}@{}}
        \toprule
        & \multicolumn{2}{c}{Means} & \multicolumn{2}{c}{Variances}\\
        \cmidrule(r){2-3}\cmidrule(l){4-5}
        {Net Gain Statistics:} & 
        $\E(G_1)$ & $\E(\oft{G}_1)$ & $\var(G_1)$ & $\var(\oft{G}_1)$\\
        {Cost Statistics:} &
        $\E(C_1)$ & $\E(\oft{C}_1)$ & $\var(C_1)$ & $\var(\oft{C}_1)$\\
        {Time Statistics:} & 
        $\E(T_1)$ & $\E(\oft{T}_1)$ & $\var(T_1)$ & $\var(\oft{T}_1)$\\
        \bottomrule
\end{tabular}
\caption[Common Statistics for Solitary Optimization]{Common statistics
used in optimization of solitary agent behavior.}
\label{tab:common_stats}
\end{table}
%
However, other statistics like $\E(G^N/T^N)$ (\ie, average gain per unit
time) or $\E( (G^N + C^N)/C^N )$ (\ie, average efficiency) for all $N
\in \N$ could also be relevant.
\index{stochasticity!statistics!skewness|(}Economists
\citep[\eg,][]{CDHP97,Kane82,KSY93,Lai91,Tsiang72} might argue that the
skewness\footnote{\index{skewness|see{stochasticity, statistics,
skewness}}\index{stochasticity!random variable|(}For a random variable
$X$, its \emph{skewness} is a measure of the
\index{symmetry|see{stochasticity, distributions,
symmetry}}\index{stochasticity!distributions!symmetry}symmetry of its
(\aimention{Henri L. Lebesgue}Lebesgue) probability density $f_X$. The
standard definition of skewness is $\E((X - \E(X))^3)/\std(X)^3$. Note
that this is a scaled version of the third
\index{stochasticity!statistics!central moments}central
moment.\index{stochasticity!random variable|)}} of each of these random
variables would be a reasonable statistic to study because it may be
desirable to have random variables that are distributed asymmetrically
(\eg, net gains that are more often high than low)\footnote{This might
be called \index{skewness preference}skewness preference. It is also
desirable to optimize skewness simply to prevent deleterious
asymmetry.}.\index{stochasticity!statistics!skewness|)} Of course, any
one of these statistics may not capture all relevant objectives of a
problem. For example, it may be desirable to maximize both $\E(G_1)$ and
$-\E(T_1)$ (\ie, minimize $\E(T_1)$); however, it may not be possible to
accomplish both of these simultaneously. Therefore, here we discuss the
construction of compound objectives that allow for optimization with
respect to multiple criteria. 

\index{multiobjective optimization|see{optimality, multiobjective
optimization}}%
\index{compound objectives|see{optimality, multiobjective
optimization}}%
\index{optimality!compound objectives|see{optimality, multiobjective
optimization}}%
\index{optimality!multiobjective optimization|(}Take a problem with $m
\in \N$ relevant optimization objectives. For all objective functions to
be minimized, replace the function with its additive or multiplicative
inverse (\ie, replace a function $f$ with the function $-f$ or, for
functions with strictly positive or strictly negative ranges, $1/f$);
therefore, the ideal objective is to maximize all $m$ functions. Collect
these $m$ objective functions into $m$-vector $\v{x}$ where $\v{x} =
\{x_1,x_2,\dots,x_m\}$. Use the weighting vector $\v{w} \in
\R_{\geq0}^m$ with $\v{w} = \{w_1,w_2,\dots,w_m\}$ to represent the
relative value of each of these objectives. Therefore, the compound
objective functions
%
\begin{equation}
        w_1 x_1 + w_2 x_2 + \cdots + w_m x_m
        \quad \text{ or } \quad
        \min\{ w_1 x_1, w_2 x_2, \dots, w_m x_m \} 
        \label{eq:compound_optimization_objectives}
\end{equation}
%
represent different ways to combine all $m$ objectives.
\index{optimality!multiobjective optimization!linear combination}The
former of these two compound objectives is a linear combination of
statistics (\ie, $\v{w}^\T \v{x}$), and an optimal behavior for this
function will be \index{Pareto optimal|see{optimality, Pareto
efficient}}\index{Pareto efficient|see{optimality, Pareto
efficient}}\index{optimality!Pareto optimal|see{optimality, Pareto
efficient}}\index{optimality!Pareto efficient}\aimention{Vilfredo
Pareto}Pareto efficient\footnote{\index{optimality!Pareto
efficient|(indexdef}To be \emph{\aimention{Vilfredo Pareto}Pareto
efficient} or \emph{\aimention{Vilfredo Pareto}Pareto optimal} means
that any deviation that yields an increase in one objective function
will also result in a decrease in another objective function.
\aimention{Vilfredo Pareto}Pareto optimal solutions characterize
tradeoffs in optimization objectives. If deviation from some behavior
will increase all objective functions, then that behavior cannot be
\aimention{Vilfredo Pareto}Pareto efficient.  The set of all
\aimention{Vilfredo Pareto}Pareto efficient solutions is called the
\index{frontier|see{optimality, Pareto efficient}}\index{Pareto
frontier|see{optimality, Pareto efficient}}\index{optimality!Pareto
frontier|see{optimality, Pareto efficient}}\emph{\aimention{Vilfredo
Pareto}Pareto frontier}.\index{optimality!Pareto efficient|)indexdef}}
with respect to the $m$ objective functions. Maximization of the latter
of these two compound objectives represents a
\index{optimality!multiobjective optimization!maximin}\emph{maximin}
optimization problem.  \index{Lagrange multiplier method|see{optimality,
Lagrange multiplier method}}\index{optimality!Lagrange multiplier
method}\aimention{Joseph-Louis Lagrange}Lagrange multiplier methods
(\ie, \index{Kuhn-Tucker conditions|see{optimality, KKT
conditions}}\index{KKT conditions|see{optimality, KKT
conditions}}\index{Karush-Khun-Tucker (KKT) conditions|see{optimality,
KKT conditions}}\aimention{William Karush and Harold W. Kuhn and Albert
W. Tucker}\acro[\index{optimality!KKT
conditions}Karush-Khun-Tucker~(KKT)]{KKT}{\index{optimality"!KKT
conditions"|indexglo}Karush-Kuhn-Tucker} conditions) \citep{Bertsekas95}
can be used to study the optimal solutions to both forms in
\longref{eq:compound_optimization_objectives}.%
\index{optimality!multiobjective optimization|)}

\subsection{Optimization Constraints}%
\label{sec:optimization_constraints}%
\index{optimization constraints|(}

In a given foraging problem, it is not necessarily the case that all
modeled behaviors are applicable or even possible. That is, optimization
analysis must be considered with respect to a set of feasible behaviors.
The following are some examples of constraints that have been found in
the literature; suggestions for how those constraints could be
implemented in this model are also given.
%
\begin{description}
        \item\emph{Time Constraints:} \index{optimization
                constraints!time}The economics-inspired graphical
                foraging model of \citet{Rapport71} considers level
                \emph{indifference curves} of an energy function. Each
                of these curves represents a set of combinations of prey
                where each combination returns the same energetic gain
                to the forager.  \Citeauthor{Rapport71} then assumes
                that the forager has a finite lifetime and surrounds all
                prey combinations that can be completed in this time
                with a boundary called the \emph{consumption
                frontier}\footnote{The consumption frontier is a Pareto
                frontier. Diets on this frontier return the greatest
                gain for their foraging time.}. The optimal diet
                combination is the point of tangency between the
                consumption frontier and some indifference curve. In
                other words, this is the combination of prey items that
                returns the highest energetic gain for the given finite
                lifetime. We can quantify this idea by maximizing
                $\E(G(t))$ subject to the constraint $t \leq
                \overline{T}$ where $\overline{T} \in \R_{>0}$. Because
                \citeauthor{Rapport71} gives a qualitative explanation
                for the observations in \citet{Murdoch69}, the
                analytical application of our model with this time
                constraint could give a quantitative explanation. 
        \item\emph{Nutrient Constraints:} \index{optimization
                constraints!nutrients}\citet{Pulliam75} optimizes a
                point gain per unit time function similar in form to
                $\E(\oft{G}_1)/\E(\oft{T}_1)$, but the notion of
                nutrient constraints is added. That is, there are $m \in
                \N$ nutrients and all tasks of type $i \in
                \{1,2,\dots,n\}$ return quantity $\rho_{ij}$ of nutrient
                $j \in \{1,2,\dots,m\}$. \Citeauthor{Pulliam75} then
                calls $M_j \in \R_{\geq0}$ a minimum amount of nutrient
                $j$ that must be returned from processing. The goal is
                to maximize the rate of point gain while maintaining
                this minimum nutrient level. These nutrient constraints
                could be added to our model as well. As
                \citeauthor{Pulliam75} notes, under these constraints,
                optimal behaviors often include partial preferences. In
                the unconstrained classical \ac{OFT} problem, it is
                sufficient for optimality to either process all or none
                of tasks of a particular type; however, with nutrient
                constraints it may be necessary for optimality that only
                a fraction of the encountered tasks of a certain type be
                processed\footnote{In \longref{ch:optimization_results},
                we generalize the classical \ac{OFT} result to show that
                over a closed interval of preference probabilities,
                sufficiency is associated with the endpoints. The
                results of \citet{Pulliam75} effectively make that
                interval a function of nutrition requirements; under
                these constraints, partial preferences may be necessary
                for optimality.}.
        \item\emph{Encounter-Rate Constraints:} \index{optimization
                constraints!encounter-rate}\citet{GS83} and \citet{PP06}
                explore the optimization of a point gain per unit time
                function as well; however, the impact of speed choice on
                imperfect detection\index{speed-accuracy tradeoff|(} is
                also introduced.  That is, with perfect detection, an
                increase in speed will most likely come with an increase
                in encounter rate with tasks of every type. However,
                when detection errors can occur, the relationship
                between encounter rate and speed may be arbitrarily
                nonlinear. If this exact relationship is not known, it
                may be sufficient to restrict search speed to a range
                where detection is reliable. If the impact of search
                speed were added to our model (\eg, if encounter-rate
                was parameterized by speed), this restriction could be
                modeled as constraints on search speed. The resulting
                optimal behavior would include a search speed that
                provides the optimal encounter rates subject to
                imperfect detection.\index{speed-accuracy tradeoff|)}
\end{description}
%
Any optimization function of a form in
\longref{eq:compound_optimization_objectives} subject to a finite number
of equality or non-strict inequality constraints\footnote{A
\emph{strict} inequality constraint uses $<$ or $>$; therefore, a
\emph{non-strict} or \emph{weak} inequality constraint uses $\leq$ and
$\geq$.} may be analyzed with \index{optimality!Lagrange multiplier
method}\aimention{Joseph-Louis Lagrange}Lagrange multiplier methods.
Therefore, in principle, a wide range of constrained optimization
problems can be studied.  \index{optimization constraints|)}

\subsection{Impact of Function Choice on Optimal Behaviors}
\label{sec:optimization_choice_effects}

As discussed in \longref{sec:oft_long_term_rate}, classical \ac{OFT}
results come from maximizing the long-term rate of gain (\eg,
$\E(\oft{G}_1)/\E(\oft{T}_1)$). This choice follows from the argument of
\citet{PPC77} that optimizing this long-term rate synthesizes the two
extremes, energetic maximization and time minimization, of a general
model of foraging given by \citet{Schoener71}. This rate approach is
taken by \citet{Pulliam75} whose quantitative results show that the
optimal diet predicted by a rate maximizer depend only on the encounter
rates with prey types in the diet. However, \citet{Rapport71} focusses
only on gain maximization (in finite time) and shows that the optimal
diet depends on encounter rates with all prey types. These two results
are very different, and the only justification for using the first
result follows from a purely intuitive argument from \citet{PPC77}.
However, the result from \citeauthor{Rapport71} is entirely valid from a
perspective of the foundational work of \citeauthor{Schoener71}.
Therefore, it is clear that one optimization criterion will not fit all
problems. Clearly, is important to investigate other functions that may
be more appropriate for specific problems. 

\section{Classical OFT Approach to Optimization}%
\label{sec:oft_approach_optimization}%
\index{solitary agent model!classical analysis!optimization|(}

As discussed by \citet{SC82}, classical \ac{OFT} approaches optimization
from two perspectives which are both based on evolutionary arguments.
The first analyzes behaviors that optimize of the asymptotic limit of
rate of net gain. The second assumes the agent must meet some energetic
requirement and maximizes its probability of success. The former, which
we describe in \longref{sec:oft_long_term_rate}, is called \emph{rate
maximization}, and the latter, which we describe in
\longref{sec:oft_risk_sensitivity}, is described as being \emph{risk
sensitive}. Both approaches develop optimal static behaviors for the
solitary agent.

\subsection{Maximization of Long-Term Rate of Net Gain}%
\label{sec:oft_long_term_rate}%
\index{long-term average rate of net gain|(}%
\index{rate maximization|(}

\index{long-term average rate of net gain!justification|(}%
In biological contexts, it is expected that natural selection will favor
foraging behaviors that provide greater future reproductive success, a
common surrogate for \aimention{Charles R. Darwin}Darwinian fitness. So,
functions mapping specific behaviors to quantitative measures of
reproductive success can be optimized to predict behaviors that should
be maintained by natural selection. \Citet{Schoener71} defines such a
model, and while quantities in the model are too difficult to define for
most cases, behaviors predicted by the model fall on a continuum from
foraging time minimizers (when energy is held constant) to energy
maximizers (when foraging time is held constant). In other words,
behaviors should be excluded if there exists another behavior that has
both a higher energy return and a lower time. \Citet{PPC77} argue that
the \emph{rate} of net energy intake is the most general function to be
maximized as it captures both extremes on the \citeauthor{Schoener71}
continuum by asserting an upward pressure on energy intake and a
downward pressure on foraging time. This will allow a forager to achieve
its energy consumption needs while also leaving it enough time for other
activities such as reproduction and predator avoidance. This
interpretation is only valid over the space of behaviors with positive
net energetic intake.  For example, rate maximization puts an upward
pressure on foraging time for behaviors that return negative net
energetic intake. This is not recognized by \citeauthor{PPC77}, and the
continuum of behaviors described by \citeauthor{Schoener71} explicitly
exclude these time maximizers. However, from a survival viewpoint, it
makes sense that foragers facing a negative energy budget should
maximize time foraging. Therefore, rate maximization encapsulates two
conditional optimization problems; it trades off net gain and total time
in a way that is dependent upon energy reserves.%
\index{long-term average rate of net gain!justification|)}%

\index{long-term average rate of net gain!limit|(}%
The rate of net energy intake can be defined in different ways. Using
the terms from \longref{ch:model}, it could be defined as $\oft{G}(t)/t$
or $\E( \oft{G}(t) )/t$ for any $t \in \R_{\geq0}$ or
$\oft{G}^N/\oft{T}^N$ or $\E( \oft{G}^N/\oft{T}^N )$ for any $N \in \N$.
However, \citeauthor{PPC77} also argue that rates should be calculated
over the entire lifetime of the forager.  Thus, rather than taking a
particular $t \in \R_{\geq0}$ or $N \in \N$, the asymptotic limits of
these ratios should be taken. Conveniently,
\longref{eq:oft_payoff_long_rate} shows that all of these limits are
equivalent. By \longref{eq:oft_payoff_long_rate_equiv}, 
%
\begin{equation} 
        \begin{split}
        \frac{\E\left(\oft{G}_1\right)}{\E\left(\oft{T}_1\right)} 
        &= 
        \frac
        {\E\left(\oft{G}^{N_*}\right)}{\E\left(\oft{T}^{N_*}\right)}
        =
        \aslim\limits_{N \to \infty} \frac{\oft{G}^N}{\oft{T}^N} 
        =
        \lim\limits_{N \to \infty} 
        \E\left(\frac{\oft{G}^N}{\oft{T}^N}\right)\\
        &=
        \frac{\E\left(\oft{G}(t_*)\right)}{\E\left(\oft{T}(t_*)\right)}
        = \aslim\limits_{t \to \infty} \frac{\oft{G}(t)}{t} 
        = \lim\limits_{t \to \infty} 
        \frac{\E\left( \oft{G}(t) \right)}{t}\\
        \end{split}
        \label{eq:oft_payoff_long_rate_summary} 
\end{equation}
%
for any $t_* \in \R_{>0}$ and $N_* \in \N$. For this reason, the ratio
of expectations $\E(\oft{G}_1)/\E(\oft{T}_1)$ has received significant
interest in classical \ac{OFT} \citep[\eg,][]{HouMc99,SC82,SK86}. We
call this ratio the \emph{long-term (average) rate of net gain}. Note
that by \longref{eq:RoE_equivalence} this ratio plays an identical role
in our analysis approach when we consider the asymptotic case.%
\index{long-term average rate of net gain!limit|)}%

\subsubsection{Opportunity Cost and Pareto Optimality}\aimention{Vilfredo
Pareto}

%\citet[ch.~4]{HouMc99} 
\index{opportunity cost|see{long-term average rate of net gain,
opportunity cost}}%
\index{long-term average rate of net gain!opportunity cost|(}%
\Citet{HouMc99} provide an interesting interpretation of
$\E(\oft{G}_1)/\E(\oft{T}_1)$. They define constant $\oft{\gamma}^* \in
\R$ to be the maximum value of $\E(\oft{G}_1)/\E(\oft{T}_1)$ (\ie, the
long-term rate of net gain) over the set of feasible agent behaviors.
They then treat rate $\oft{\gamma}^*$ as a factor converting time spent
between encounters to maximum points possible from that time. Therefore,
$\oft{\gamma}^*$ converts time into its equivalent \emph{opportunity
cost} (\ie, gain paid per unit time). They show that the behavior that
maximizes 
%
\begin{equation}
        \E\left(\oft{G}_1 - \oft{\gamma}^* \oft{T}_1\right)
        \label{eq:HouMc99_approach}
\end{equation}
%
will also be the behavior that achieves the maximum long-term rate of
gain $\oft{\gamma}^*$. So, maximizing the long-term rate of gain is
equivalent to maximizing the per-cycle \emph{gain} after being
discounted by the opportunity cost of the cycle time\footnote{There is a
related result by \citet{ES84} that predicts the optimal behavior on
simultaneous encounters. This is described by both \citet{HouMc99} and
\citet{SK86}, and \citeauthor{HouMc99} show this simultaneous encounter
result to follow from the opportunity cost result.}. Solving for this
behavior can be done analytically only if $\oft{\gamma}^*$ is known, and
so the method of \citeauthor{HouMc99} numerically solves for the optimal
behavior using iteration, which could be a weakness of this approach.
However, it demonstrates an important interpretation of
$\E(\oft{G}_1)/\E(\oft{T}_1)$ as the opportunity cost of time. Not
surprisingly, this also shows that the behavior that maximizes the
long-term rate of gain is \aimention{Vilfredo Pareto}Pareto optimal with
respect to maximization of $\E(\oft{G}_1)$ and (maximization)
minimization of $\E(\oft{T}_1)$ when $\oft{\gamma}^* > 0$
($\oft{\gamma}^* < 0$); that is, this optimal behavior represents a
particular tradeoff between net gain and total time. This
\aimention{Vilfredo Pareto}Pareto interpretation casts $\oft{\gamma}^*$
as the relative importance of minimizing time, which is consistent with
notion of opportunity cost\footnote{When $\oft{\gamma}^* < 0$, the
relative importance of minimizing time is \emph{negative}, which
indicates that $|\oft{\gamma}^*|$ is the relative importance of
\emph{maximizing} time (\ie, an opportunity \emph{gain}).}. The
numerical approach to finding $\oft{\gamma}^*$ and the corresponding
optimal behavior is equivalent to sliding along a continuum of
\aimention{Vilfredo Pareto}Pareto efficient solutions (\ie, tradeoffs of
net gain and total time).%
\index{long-term average rate of net gain!opportunity cost|)}%

\subsubsection{Equilibrium Renewal Process as an Attractive Alternative}

\index{equilibrium process rate of net gain|see{equilibrium renewal
process rate of net gain}}%
\index{equilibrium renewal process rate of net gain|(}%
\Citet{Cha73} note that it is desirable to derive the \emph{equilibrium
renewal process} rate of net gain. That is, introduce a $T_1 \in
\R_{>0}$ and redefine the process to start after $T_1$ foraging time has
past. Hence, runtime $t$ represents the length of the interval
immediately after time $T_1$, and so quantity of interest to
\citeauthor{Cha73} is $\E(G(t))/t$, which represents the average rate of
net gain returned to an agent when the agent is in equilibrium with its
environment (\ie, after the decay of any initial transients). However,
they point out that this rate is only known for such a process if it is
additionally assumed that the net gain on each \ac{OFT} cycle is
independent of the total time of each \ac{OFT} cycle (in particular, the
processing time of each cycle). In that case, $\E(G(t))/t$ can also be
expressed as the ratio $\E(\oft{G}_1)/\E(\oft{T}_1)$. Unfortunately, it
is rare that net gain and processing time will be independent in a
practical system. Analytical results are not available otherwise.  For
this reason, when $\E(\oft{G}_1)/\E(\oft{T}_1)$ is used it is usually
assumed to be a limiting case (\ie, a rate over a long time rather than
a short-term rate after a long time).%
\index{equilibrium renewal process rate of net gain|)}%

\subsubsection{Graphical Interpretation of Rate Maximization}%
\index{long-term average rate of net gain!graphical optimization|(}%

When an agent is only free to choose its (average) processing times, the
tasks are said to occur in \emph{patches} or to be \emph{patchily
distributed} \citep{SK86}. Take such a case with a single task type and
no search or processing costs (\ie, $n=1$, $c^s=c_1=0$, $p_1=1$, and
$\tau_1 \in \R_{\geq0}$). \Citet{SK86} show that this problem has an
insightful graphical solution. Consider
\longref{fig:oft_rate_maximization}. 
%
%\begin{figure}[!ht]\centering
\begin{figure}\centering
        \begin{picture}(140,109)(0,0)
                \put(70, 54.5){%
                \makebox(0,0){\scalebox{0.9}{\framebox(87.5,95){
                \shortstack[c]%
                {$\E(\oft{G}_1) = g_1(\tau_1)$\\%
                $\E(\oft{T}_1) =  \tau_1 + \frac{1}{\lambda}$\\%
                $\oft{\gamma} \triangleq \frac{g_1(\tau_1)}%
                {\tau_1 + \frac{1}{\lambda}}$\\%
                $\lambda = \lambda_1$\\%
                $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}}}
        \end{picture}
        \begin{picture}(160,109)(-33,-21)
                % Horizontal Axis
                \thicklines
                \put(-30, 0){\vector(1, 0){144}}
                \put(115, 0){\makebox(0, 0)[l]{$\tau_1$}}
                % Vertical Axis
                \thicklines
                \put(0, -16){\vector(0, 1){90}}
                \put(0, 75){\makebox(0, 0)[b]{$g_1(\tau_1)$}}
                % Rate Line
                \thinlines
                \put(-24, 0){\line(2,1){138}}
                \put(20, 22){\circle*{5}}
                % Maximal Rate (Slope)
                \thinlines
                \put(74,67){\line(1,0){36}}
                \put(74,49){\line(0,1){18}}
                \put(76,65.5){\makebox(0, 0)[lt]{$\oft{\gamma}^*$}}
                % Curve, starting origin and moving CW
                \thinlines
                \qbezier(0,0)(0,12)(20,22)
                \qbezier(20,22)(56,40)(78,40)
                \qbezier(78,40)(97,40)(114,30)
                %\cbezier(20,22)(56,40)(90,40)(114,30)
                % Max and mins
                \thicklines
                \put(-4,40){\line(1,0){8}}
                \put(20,-4){\line(0,1){8}}
                \thinlines
                \put(-5,40){\makebox(0,0)[r]{$g_1^*$}}
                \put(0,40){\dashbox{3}(78,0){}}
                \put(20,-5){\makebox(0,0)[t]{$t^*$}}
                %\put(20,0){\dashbox{3}(0,20){}}
                % Origin strategy
                \thicklines
                \put(-24,-4){\line(0,1){8}}
                \thinlines
                \put(-24,-5){\makebox(0, 0)[t]{$\frac{-1}{\lambda}$}}
        \end{picture}
        \caption[Visualization of Classical OFT Rate Maximization]{Rate
        maximization in classical \ac{OFT}. It is assumed that $n=1$,
        $c^s = 0$, and $c_1 = 0$. The constraint that $p_1 = 1$ is also
        applied. The optimal processing time is denoted $t^*$, and the
        corresponding maximal rate is denoted $\oft{\gamma}^*$ and shown
        as a slope of a tangent line.}
        \label{fig:oft_rate_maximization}
\end{figure}
%
The $g_1(\tau_1)$ function is plotted with respect to feasible choices
of $\tau_1$ and a mark is made at the point $(-1/\lambda,0)$. For any
$\tau_1$, the corresponding long-term rate of gain is the slope of a
line that connects points $(-1/\lambda,0)$ and $(\tau_1,g(\tau_1))$.
Therefore, the optimal $\tau_1$ (shown as $t^*$) is the one that
corresponds with the line with the maximal slope, and that slope will be
the maximal long-term rate of gain (shown as $\oft{\gamma}^*$). In
\longref{sec:alternate_rate_tradeoffs}, we show how this graphical
interpretation can be extended to the general case\footnote{We show this
interpretation using our approach to defining the relevant statistics of
the model; however, our method can also be applied to the classical
\ac{OFT} statistics in an obvious way (\ie, with little more than a
change of notation).} (\ie, with multiple types, costly searching and
processing, and tasks that may or may not be patchily distributed).

\index{MVT|see{marginal value theorem}}%
Several conclusions can be drawn from
\longref{fig:oft_rate_maximization}. For differentiable functions with
$g_1(0)=0$ and $g_1'(0)>0$, the optimal processing time $t^*$ must be
such that $g_1'(t^*)$ is equal to the long-term rate of gain. In
particular, if $g_1$ is a concave function, then this line will be the
\emph{unique} tangent line that crosses $(0,1/\lambda)$.
Rate-maximization for the classical \ac{OFT} model is said to follow the
\emph{\acro[\index{marginal value theorem (MVT)|indexdef}marginal value
theorem~(MVT)]{MVT}{\index{marginal value theorem
(MVT)"|indexglo}marginal value theorem}} \citep{Cha76,Cha73}. This means
that the average time an agent processes patchily distributed tasks of a
certain type is the time when the average rate of point gain for the
task type drops to the average rate of point gain for the environment.
That is, processing should continue until the marginal return from the
next instant of processing is less than the environmental average rate
of gain\footnote{This interpretation is really only accurate for a
deterministic agent model. In the general stochastic agent model, the
\ac{MVT} need only be observed in the first-order statistics of the
gains and processing times.}.%
\index{long-term average rate of net gain!graphical optimization|)}%
\index{long-term average rate of net gain|)}
\index{rate maximization|)}

\subsection{Minimization of Net Gain Shortfall}%
\label{sec:oft_risk_sensitivity}%
\index{risk minimization|see{risk sensitivity}}%
\index{risk sensitivity|(}\index{mean-variance analysis (MVA)|(}

Because rate maximization depends only on first-order statistics, it
disregards the \index{stochasticity!statistics!standard
deviation}standard
deviation\footnote{\index{stochasticity!statistics!standard
deviation|(indexdef}For random variable $X$, the standard deviation
$\std(X)$ is $\sqrt{\var(X)}$ (\ie, the square root of the
variance).\index{stochasticity!statistics!standard deviation|)indexdef}}
of random variables in the model. For example, an agent with a behavior
that maximizes its long-term rate of net gain may bypass frequently
encountered tasks with small gains regardless of any survival needs.
\index{gain threshold|see{gain success threshold}}\index{survival
threshold|see{gain success threshold}}\index{threshold|see{gain success
threshold}}\index{success threshold|see{gain success
threshold}}\index{gain success threshold}However, if the agent must meet
a net gain requirement in finite time, it may be beneficial to decrease
mean net gain if that decrease also comes with an decrease in the
uncertainty of returns. 

\subsubsection{Maximization of Reward-to-Variability Ratio}

\Citet{SC82} introduce a risk-sensitive agent model and an optimization
approach that maximizes the \index{probability of success}probability of
success. Consider a solitary agent that must acquire some minimal net
gain \index{gain success threshold}$\oft{G}^T$ by a time $\oft{T} \in
\R_{\geq0}$. Call $\oft{\mu}$ the expectation and $\oft{\sigma}$ the
standard deviation of net gain acquired by $\oft{T}$ for some given
behavior. The method states that the desired risk-sensitive behavior
should maximize the objective
%
\begin{equation}
        \frac{\oft{\mu} - \oft{G}^T}{\oft{\sigma}}
        \label{eq:oft_zscore}
\end{equation}
%
\index{location|see{stochasticity, distributions, location-scale
family}}\index{scale|see{stochasticity, distributions, location-scale
family}}\index{stochasticity!distributions!location-scale family|(} If
the net gain random variable is \index{location-scale|see{stochasticity,
distributions, location-scale family}}
location-scale\footnote{\index{stochasticity!distributions!location-scale
family|indexdef}A family of distribution functions $\Omega$ is called
\emph{location-scale} if there exists some $F \in \Omega$ such that for
all $F_1 \in \Omega$, there exists a
\index{stochasticity!distributions!location|see{stochasticity,
distributions, location-scale}}\emph{location} $m \in \R$ and
\index{stochasticity!distributions!scale|see{stochasticity,
distributions, location-scale}}\emph{scale} $s \in \R_{>0}$ with $F_1(x)
= F( (x-m)/s )$. A random variable is location-scale if its distribution
comes from such a family. This idea of a two parameter family of
distribution functions comes from \citet{RS70}, and this definition of
such a class of functions is due to \citet{Bawa75}; however,
\citet{Meyer87} gives an equivalent definition.  Examples of
location-scale distributions are the
\index{stochasticity!distributions!normal}normal,
\index{stochasticity!distributions!exponential}exponential,
\index{stochasticity!distributions!uniform}, and
\index{stochasticity!distributions!double exponential}double exponential
distributions.} with identical
\index{stochasticity!statistics!skewness}skewness for all choices of
location and scale\footnote{Location-scale distributions with mean
locations and standard deviation scales will naturally have this
property.}, the behavior that maximizes \longref{eq:oft_zscore} will
also minimize the probability that the net gain is less than the
$\oft{G}^T$ threshold\footnote{This is a sufficient condition; however,
it is not necessary. Investment theoretic consequences of location-scale
distributions are given by \citet{Bawa75} and \citet{Meyer87}. The
multivariate case is handled by \citet{Chamberlain83} and
\citet{OR83}.}. \index{gain success threshold|(}In other words, if the
agent is said to be \emph{successful} when its net gain meets or exceeds
$\oft{G}^T$, then the optimal behavior will maximize the probability of
success\footnote{This result can be generalized slightly by considering
the class of distributions where a monotonic transformation (\ie,
continuously differentiable with non-negative derivative everywhere) of
random variables is location-scale. The
\index{stochasticity!distributions!log-normal}log-normal distribution
belongs to this more general class \citep{Bawa75}.}.\index{gain success
threshold|)}

\paragraph{Location-Scale Justification:} By the
\index{CLT|see{stochasticity, central limit theorem}}\index{central
limit theorem (CLT)|see{stochasticity, central limit
theorem}}\emph{\acro[\index{stochasticity!central limit theorem
(CLT)}central limit theorem~(CLT)]{CLT}{\index{stochasticity"!central
limit theorem (CLT)"|indexglo}central limit theorem}}, if the net gain
is a sum of \iid{}\ random variables (\eg, individual cycle gains), the
probability distribution of the net gain will approach a
\index{stochasticity!distributions!normal}normal
distribution\footnote{\index{stochasticity!distributions!normal|(indexdef}A
\emph{normal} or \index{normal distribution|see{stochasticity,
distributions, normal}}\index{Gaussian distribution|see{stochasticity,
distributions, normal}}\emph{\aimention{Carl Friedrich Gau\ss}Gaussian}
random variable $X$ with mean $\mu$ and standard deviation $\sigma$ has
(\aimention{Henri L. Lebesgue}Lebesgue) probability density $f_X(x) =
1/( \sigma \sqrt{2 \pi}) \exp( -(x-\mu)^2/(2 \sigma^2) )$. Normal random
variables are location-scale with location $\mu$ and scale $\sigma$ and
are \index{stochasticity!distributions!symmetry}symmetric about their
mean (\ie, they have zero
skewness).\index{stochasticity!distributions!normal|)indexdef}} as the
number of elements in the sum increases. Therefore, it may be reasonable
(\eg, consider $\oft{G}^N$ as $N \to \infty$) to assume that net gains
are \index{stochasticity!distributions!normal}normally distributed or at
least location-scale with location-scale invariant skewness. In this
case, the behavior that maximizes \longref{eq:oft_zscore} will certainly
maximize the probability of
success.\index{stochasticity!distributions!location-scale family|)}

\paragraph{Analogous Results from Economics:} \citet{SK86} call this the
\index{z-score@$z$-score|see{reward-to-variability
ratio}}\emph{$z$-score model}; however, it is well-known in economics
that this method was initially developed by \citet{Sharpe66} for
application to optimal portfolio selection. \Citeauthor{Sharpe66} calls
\longref{eq:oft_zscore} a \index{reward-to-variability
ratio|indexdef}\emph{reward-to-variability ratio}\footnote{This is also
known as the \index{Sharpe ratio|see{reward-to-variability
ratio}}\emph{\citeauthor{Sharpe66} ratio}, which is named after the
\aimention{Alfred B. Nobel}Nobel laureate who developed it.}.  While
economists realize that return distributions need not be
\index{stochasticity!statistics!normal}normally distributed (\eg,
symmetric about the mean) for the reward-to-variability ratio to
minimize risk, \citet{SK86} depend on
\index{stochasticity!distributions!normal}normality to justify their
claims \citep[p.~134]{SK86}. Assuming
\index{stochasticity!distributions!normal}normality of returns may be
far too restrictive. In fact, it is desirable that returns are skewed so
that the mass is concentrated on higher gains (\ie, not symmetric and
therefore not \index{stochasticity!distributions!normal}normal).
Therefore, by depending on consistent skewness rather than symmetry, the
economic argument of reward-to-variability maximization is not only more
general but also more convincing than the argument of \citeauthor{SK86}.

\paragraph{Links to Risk-Sensitive Dynamic Optimization:} \index{dynamic
optimization}An \index{ex post performance measure@\expost{} performance
measure}\expost{} version of the \index{reward-to-variability
ratio}reward-to-variability ratio is described by
\citet{Sharpe94}\footnote{\Citet{Sharpe94} also discusses other related
ratios which may have applicability to stochastic optimization of static
behavior.}, which is typically used for measuring past performance.
However, there may some opportunity to use this \expost{} ratio for
dynamic optimization to derive an optimal control similar to the
bang-bang\footnote{\index{bang-bang control|(indexdef}A \emph{bang-bang}
control switches an actuator from one extreme to the other. In this
context, the variance at each small time period could be chosen by the
agent to be either its lowest value or its highest
value.\index{bang-bang control|)indexdef}} control described by
\citet{SK86} and \citet{McNamara83} and based on the $z$-score model.

\subsubsection{Graphical Interpretation of Risk Minimization}%
\index{reward-to-variability!graphical optimization|(}

Again, consider an agent described by the model in \longref{ch:model}
with tasks that are patchily distributed. Assume that there is a single
task type and no search or processing costs (\ie, $n=1$, $c^s=c_1=0$,
$p_1=1$, and $\tau_1 \in \R_{\geq0}$), but assume that the agent must
acquire a net gain of $\oft{G}^T$ in $\oft{T}$ time.  \Citet{SC82} use
this scenario to illustrate reward-to-variability maximization, as shown
in \longref{fig:oft_risk_sensitivity}. 
%
%\begin{figure}[!ht]\centering
\begin{figure}\centering
        \begin{picture}(150,181)(0,0)
                \put(75, 90.5){\makebox(0,0){
                \framebox(110,55){
                \shortstack[c]%
                {$t_a < t_b < t^* < \oft{T}$\\
                $\oft{\rho}%
                \triangleq\frac{\oft{\mu}-\oft{G}^T}{\oft{\sigma}}$\\%
                $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}}
        \end{picture}
        \begin{picture}(117,181)(2,-17)
                \thicklines
                % Horizontal Axis
                \put(5, 0){\vector(1, 0){105}}
                \put(111, 0){\makebox(0, 0)[l]{$\oft{\sigma}$}}
                % Vertical Axis
                \put(24, -15){\vector(0, 1){165}}
                \put(24, 151){\makebox(0, 0)[b]{$\oft{\mu}$}}
                % Z-score line
                \thinlines
                \put(24, 136.5){\line(2,-1){86}}
                \put(80, 108.5){\circle*{5}}
                % Maximal Z-score (slope)
                \thinlines
                \put(29,134){\line(1,0){39}}
                \put(68,134){\line(0,-1){19.5}}
                \put(67.5,132.5){\makebox(0, 0)[rt]{$\oft{\rho}^*$}}
                % Threshold Value
                \thicklines
                \put(20,136.5){\line(1,0){8}}
                \thinlines
                \put(19,136.5){\makebox(0,0)[r]{$\oft{G}^T$}}
                % Curve, starting from origin and moving CCW 
                \thinlines
                \qbezier(40,121)(30,121)(30,100)
                \qbezier(80,108.5)(55,121)(40,121)
                \qbezier(92,80)(92,102.5)(80,108.5)
                \qbezier(85,38)(92,62)(92,80)
                \qbezier(54,7)(78,14)(85,38)
                \qbezier(24,0)(30,0)(54,7)
                % Example times
                \put(53.16,9.88){\line(7,-24){1.68}}
                \put(53.16,9.88){\rotatebox{286.26}%
                {\makebox(0,0)[r]{$t_a$}}}
                \put(82.12,38.84){\line(24,-7){5.76}}
                \put(82.12,38.84){\rotatebox{343.74}%
                {\makebox(0,0)[r]{$t_b$}}}
                \put(80,108.5){\rotatebox{63.4349}%
                {\makebox(0,0)[r]{$t^*\,$}}}
                %\put(27,100){\line(1,0){6}}
                %\put(34,100){\makebox(0,0)[l]{$\oft{T}$}}
                % Max and mins
                \thicklines
                \put(92,-4){\line(0,1){8}}
                \put(20,121){\line(1,0){8}}
                \thinlines
                \put(92,-5.5){\makebox(0, 0)[t]{$\oft{\sigma}^*$}}
                \put(92,0){\dashbox{3}(0,80){}}
                \put(19,121){\makebox(0, 0)[r]{$\oft{\mu}^*$}}
                \put(24,121){\dashbox{3}(16,0){}}
        \end{picture}
        \caption[Visualization of Classical OFT Risk-Sensitive
        Solutions]{Risk sensitivity in classical \ac{OFT}. Over a time
        period of length $\oft{T}$, $\oft{\mu}$ is the expected net
        gain, $\oft{\sigma}$ is the standard deviation of the net gain,
        and $\oft{G}^T$ is an average net gain success threshold. The
        curve shows $(\oft{\mu},\oft{\sigma})$ combinations for each
        average processing time, and $t_a$ and $t_b$ are examples of
        average processing times. The optimal average processing time is
        denoted $t^*$ and the corresponding maximal
        reward-to-variability ratio is denoted by $\oft{\rho}^*$ and
        shown as the slope of a tangent line.}
        \label{fig:oft_risk_sensitivity}
\end{figure}
%
The curve shows the $(\oft{\mu},\oft{\sigma})$ combinations that result
from each choice of processing time $\tau_1$, where processing times
increase in a counter-clockwise direction. Because the gain threshold is
so high, the optimal processing time, denoted $t^*$, is biased toward a
higher-variance distribution of outcomes. In other words, the negative
energy budget forces the agent to take more risks to maximize the
probability of success. In \longref{sec:alternate_rate_tradeoffs}, we
demonstrate how this graphical solution can be used to gain insight into
the most general case of the model from
\longref{ch:model}\footnote{Again, while our solution will not be in
terms of the statistics used in classical \ac{OFT}, it can easily be
extended with little more than a change of notation. Also, to maintain
tractability, we assume \longrefs{eq:cycle_proc_gross_gain_2moment}--%
\shortref{eq:cycle_net_gain_2moment}.}.%
\index{reward-to-variability!graphical optimization|)}%
\index{mean-variance analysis (MVA)|)}\index{risk sensitivity|)}

\subsection{Criticisms of the OFT Approach}
\label{sec:oft_criticisms}

Despite its successes in making qualitative predictions that agree with
empirical data, rate maximization of the classical \ac{OFT} approach is
often criticized. \index{patch overstaying}For example, \citet{Nonacs01}
tabulates 19 cases of foragers being observed to process tasks longer
than predicted by rate maximization, which shows that observations tend
to deviate only in one direction. \Citeauthor{Nonacs01} concludes that
``something fundamental is missing'' \citep[p.~71]{Nonacs01} from the
classical \ac{OFT} approach.  However, as we discussed, rate
maximization exerts an upward pressure on time when energy budgets are
negative. Thus, the deficiency may not be in classical \ac{OFT} but in
gain models that do not properly include negative cases\footnote{A
generalization of this suggestion is given in
\longref{sec:alternate_rate_tradeoffs}.}. Additionally, the
risk-sensitive analysis of \citet{SC82} also explains this upward
pressure on time for similar reasons. However, this analysis receives
significantly less attention than the rate maximization approach even
though analogous risk-sensitive methods are used with great success in
other fields, like \ac{MPT}.

Alternate approaches choose to abandon the classical \ac{OFT} agent
model \citep[\eg,][]{BK95,BW96,CB05,HR87,HG05,IHY81,Real80,TL81}; these
approaches study risk aversion,
satisficing\footnote{\index{satisficing|(indexdef}Satisficing describes
a behavior that is suboptimal but achieves some minimum gain (\ie,
``survival of the more fit over the less fit, not necessarily of the
most fit'' \citep[p.~640]{CB05}).\index{satisficing|)indexdef}},
heuristic rules, and even propose fitting empirical data directly
without any functional justification for the resulting behavioral rules.
We value the generality and intuitive appeal of the classical \ac{OFT}
approach.  Additionally, even though it may be debatable whether it can
be used for evolutionary analysis in biology, optimization has natural
applications in engineering. Therefore, we use our generalized agent
model and build optimization objectives inspired by both classical
\ac{OFT} and its critics; these objectives should have applicability to
engineering and may answer some of the questions about the shortfalls of
the predictive power of classical \ac{OFT}.

\index{solitary agent model!classical analysis!optimization|)}

\section{Generalized Optimization of Solitary Agent Behavior}%
\label{sec:alternate_optimization_objectives}%
\index{solitary agent model!processing-only analysis!optimization|(}

Here, we discuss optimization criteria that may have utility in both
engineering and biology. As discussed in
\longref{sec:oft_approach_optimization}, the analysis of asymptotic
behaviors may be justifiable in evolutionary analysis. However, in
engineering applications, it may be more appropriate to focus on the
case where an agent's finite lifetime is determined entirely by a set
number of tasks the agent must \emph{complete}. Also, because design
takes the place of natural selection, we need to explore different
optimization objectives. Therefore, in this section, we use our
generalized solitary agent model and insights from both \ac{OFT} and
\ac{MPT} to generate new optimization objectives that may be applicable
to both biology and engineering. We discuss our approach to handling
finite lifetimes in \longref{sec:finite_task_processing}. In
\longref{sec:alternate_rate_tradeoffs}, we present functions that, like
classical \ac{OFT}, use rates to trade off between multiple optimization
objectives. In \longref{sec:alternate_pareto_tradeoffs}, we present
generalized \aimention{Vilfredo Pareto}Pareto optimal combinations of
optimization objectives that allow a design to tune the relative
importance of the objectives. Note that when we discuss standard
deviation in variance, we assume
\longrefs{eq:cycle_proc_gross_gain_2moment}--%
\shortref{eq:cycle_net_gain_2moment} for simplicity.

Some of the objectives that we define still operate on the sliding scale
of behaviors described by \citet{Schoener71}. The primary justification
in classical \ac{OFT} for using the long-term rate of gain discussed in
\longref{sec:oft_long_term_rate} is that it agrees with
\citeauthor{Schoener71} and makes predictions that agree
\emph{qualitatively} with empirical evidence \citep{Cha73,PPC77,SK86}.
However, some of our alternative objectives also agree with
\citeauthor{Schoener71} and have qualitiative predictions similar to
those from classical \ac{OFT}. Therefore, these criteria seem as
well-suited for biological application as the long-term rate of gain. In
fact, as discussed in \longref{sec:alternate_rate_tradeoffs}, the cases
where our approaches yield different predictions than the classical
\ac{OFT} approach may be interesting and help explain some of the
empirical inconsistencies with the long-term rate of gain approach
\citep[\eg,][]{BK95,HR87,Nonacs01}.

\subsection{Finite Task Processing}
\label{sec:finite_task_processing}

One disadvantage of studying behavior that is optimal with respect to
the long-term rate of gain is the necessity of justifying the use of
infinite-time (\ie, infinite-cycle) approximation. A true finite-time
optimization approach is desirable. There are several dynamic
programming approaches to time-limited agents \citep[\eg,][]{ClkMngl00,
HouMc99, MnglClk88, WaBeHaBo06} where behaviors are based on state.
While this state-based approach provides can be better for the analysis
animal behavior, it sometimes provides less intuition and is more
difficult to implement on-line in an engineered agent that may have very
limited computational abilities. Additionally, assuming that the time
horizon is known or even fixed may be unrealistic. Thus, an approach
that combines elements of both the infinite-time approach and the
time-limited approach could be useful. 

Rather than fixing the time horizon, we fix the number of tasks
processed. This is useful for modeling, for example, a situation where
an agent expends a limited resource on each processed task. Therefore,
while the total searching and processing time is finite and unknown
(\ie, random), the total number of tasks to process is fixed. So, take
$N^p \in \N$ to be a fixed number of tasks completed by an agent. Our
objective functions are based on statistics from our modeling approach.
Of course, these statistics take the total number of tasks $N^p$ as a
parameter. Therefore, it makes sense that behaviors for a small number
of total tasks may vary greatly from behaviors for a large number of
tasks. However, in all cases behaviors will not be based on state. In
particular, the behavior for the first task processed will be consistent
with the behavior for the last task processed. This allows for the
construction of behaviors that can be implemented on very simple
engineered agents.

\subsection{Tradeoffs as Ratios}
\label{sec:alternate_rate_tradeoffs}

Ratios are unique in their ability to apply pressure on one objective
conditioned on the sign of another objective. However, the derivation of
the expectation of ratios of random variables is not trivial and often
requires high-order statistics of the random variables. Despite the
drawbacks, we explore some ratios of statistics here that may be useful
when there are no other easy ways to apply the appropriate pressures for
optimization. A more general method of multiobjective optimization is
explored in \longref{sec:alternate_pareto_tradeoffs}.

For example, classical \ac{OFT} studies the optimization objective
$\E(\oft{G}_1)/\E(\oft{T}_1)$ because it is the limiting expression of
$\E(\oft{G}(t))/t$ as $t \to \infty$ and maximization of the long-term
rate of gain is desirable. The analogous expression with our assumption
of a finite number of tasks processed is $\E(G^{N^p}/T^{N^p})$, but this
expectation has poor analytical tractability. Although not ideal,
maximization of $\E(G_1)/\E(T_1)$ provides similar pressures on
behaviors and has a relatively simple analytical structure. Therefore,
it is reasonable to study behaviors that are optimal with respect to
this latter objective.

\subsubsection{Rate Maximization with Gain Threshold}

\index{rate of excess net gain|(indexdef}% 
Consider an agent that must achieve an expected net gain of at least
\index{gain success threshold}$G^T$ after $N^p$ tasks processed but must
also achieve its goal in as little time as possible. This problem has
aspects in common with both \index{rate maximization}rate maximization
and \index{risk sensitivity}risk sensitivity. Therefore, we consider the
objective function
%
\begin{equation}
        \frac{ \E(G^{N^p}) - G^T }{ \E(T^{N^p}) }
\end{equation}
%
or, equivalently,
%
\begin{equation}
        \frac{ \E(G_1) - \frac{G^T}{N^p} }{ \E(T_1) }
        \label{eq:payoff_rate}
\end{equation}
%
So, the goal of the agent is to maximize its expected \emph{excess} net
gain in as little expected time possible. This may be considered a
generalization of the long-term rate of gain optimization used in
classical \ac{OFT}.%
\index{rate of excess net gain|)indexdef}

\index{rate of excess net gain|(}%
\index{rate of excess net gain!graphical optimization|(}%
A \index{examples!graphical solutions!rate of excess net
gain|(indexdef}graphical solution to this problem is shown in
\longref{fig:rate_maximization}. This graphical approach is similar to
the time-constrained approach of \citet{Rapport71} that is described
above. 
%
%\begin{figure}[!ht]\centering
\begin{figure}\centering
        \begin{picture}(168,68)(0,0)
                \put(0, 0){\scalebox{0.8}{\framebox(210,85){
                \shortstack[c]%
                {$\mu \triangleq \E(G_1) = %
                \overline{g^p} - \overline{c^p} - \frac{c^s}{\lambda^p}%
                $\\%
                $\E(T_1) = \overline{\tau^p} + \frac{1}{\lambda^p}$\\%
                $\gamma \triangleq \frac{\mu - %
                \frac{G^T}{N^p}}{\overline{\tau^p}%
                + \frac{1}{\lambda^p}}$\\%
                $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}}
        \end{picture}\\
        \subfloat[Per-Cycle Perspective]{
        \begin{picture}(191,155)(-39,-20)
                % Horizontal Axis
                \thicklines
                \put(-30, 12){\vector(1, 0){144}}
                %\put(-30, 0){\vector(1, 0){144}}
                \put(115, 12){\makebox(0, 0)[l]%
                {$\overline{\tau^p} + \frac{1}{\lambda^p}$}}
                % Vertical Axis
                \thicklines
                \put(-12, -16){\vector(0, 1){136}}
                %\put(0, -16){\vector(0, 1){136}}
                \put(-12, 121){\makebox(0, 0)[b]{$\mu$}}
                % Rate Line
                \thinlines
                \put(-12, 92){\line(2,-1){126}}
                \put(84, 44){\circle*{5}}
                % Maximal Rate (Slope)
                \thinlines
                \put(20,76){\line(1,0){34}}
                \put(54,76){\line(0,-1){17}}
                \put(53,75){\makebox(0, 0)[rt]{$\gamma^*$}}
                % Threshold Value
                \thicklines
                \put(-16, 92){\line(1,0){8}}
                \put(-17, 92){\makebox(0, 0)[r]{$\frac{G^T}{N^p}$}}
                % Curve, starting origin and moving CW
                \thinlines
                \qbezier(0,0)(10,55)(40,55)
                \qbezier(40,55)(58.5,55)(84,44)
                %\qbezier(84,44)(92,41)(114,21)
                \qbezier(84,44)(92,41)(100,21)
                % Max and mins
                \thicklines
                \put(-16,55){\line(1,0){8}}
                \put(100,8){\line(0,1){8}}
                \thinlines
                \put(-17,55){\makebox(0, 0)[r]{$\mu^*$}}
                \put(-12,55){\dashbox{3}(52,0){}}
                \put(100,7){\makebox(0, 0)[t]%
                {$\left(\overline{\tau^p}%
                +\frac{1}{\lambda^p}\right)^*$}}
                \put(100,12){\dashbox{3}(0,9){}}
                % Origin strategy
                \thicklines
                \put(0,8){\line(0,1){8}}
                \put(-16,0){\line(1,0){8}}
                \thinlines
                \put(0,17){\makebox(0, 0)[b]%
                {$\frac{1}{\lambda^p}$}}
                \put(-17,0){\makebox(0, 0)[r]%
                {$-\frac{c^s}{\lambda^p}$}}
        \end{picture}
        \label{fig:rate_maximization_cycle}
        }
        \subfloat[Graphical Interpretation]{
        \begin{picture}(182,155)(-54,-20)
                % Horizontal Axis
                \thicklines
                \put(-30, 0){\vector(1, 0){144}}
                \put(115, 0){\makebox(0, 0)[l]{$\overline{\tau^p}$}}
                % Vertical Axis
                \thicklines
                \put(0, -16){\vector(0, 1){136}}
                \put(0, 121){\makebox(0, 0)[b]%
                {$\overline{g^p}-\overline{c^p}$}}
                % Rate Line
                \thinlines
                \put(-12, 92){\line(2,-1){132}}
                \put(84, 44){\circle*{5}}
                % Maximal Rate (Slope)
                \thinlines
                \put(20,76){\line(1,0){34}}
                \put(54,76){\line(0,-1){17}}
                \put(53,75){\makebox(0, 0)[rt]{$\gamma^*$}}
                % Threshold Value
                \thicklines
                \put(-4, 80){\line(1,0){8}}
                \put(-5, 80){\makebox(0, 0)[rt]%
                {\scalebox{0.8}{$\frac{G^T}{N^p}$}}}
                \put(-4, 92){\line(1,0){8}}
                \put(5, 90){\makebox(0, 0)[lb]%
                {$\frac{G^T}{N^p}+\frac{c^s}{\lambda^p}$}}
                \put(0, 80){\circle*{2}}
                \put(-2, 82){\circle*{1}}
                \put(-4, 84){\circle*{1}}
                \put(-6, 86){\circle*{1}}
                \put(-8, 88){\circle*{1}}
                \put(-10, 90){\circle*{1}}
                \put(-12, 92){\circle*{3}}
                \put(-14, 94){\circle*{1}}
                \put(-16, 96){\circle*{1}}
                \put(-18, 98){\circle*{1}}
                \put(-20, 100){\circle*{1}}
                \put(-22, 102){\circle*{1}}
                \put(-24, 104){\circle*{1}}
                \put(-26, 106){\circle*{1}}
                \put(-28, 108){\circle*{1}}
                \put(-30, 110){\circle*{1}}
                \put(-32, 112){\circle*{1}}
                \put(-34, 114){\circle*{1}}
                \put(-36, 116){\circle*{1}}
                \put(-38, 118){\vector(-1,1){1}}
                \put(-28,108){\line(0,-1){6}}
                \put(-28,102){\line(1,0){6}}
                \put(-29,105){\makebox(0, 0)[r]
                {\scalebox{0.8}{$-c^s$}}}
                % Curve, starting origin and moving CW
                \thinlines
                \qbezier(0,0)(10,55)(40,55)
                \qbezier(40,55)(58.5,55)(84,44)
                %\qbezier(84,44)(92,41)(114,21)
                \qbezier(84,44)(92,41)(100,21)
                % Max and mins
                \thicklines
                \put(-4,55){\line(1,0){8}}
                \put(100,-4){\line(0,1){8}}
                \thinlines
                \put(-5,54){\makebox(0, 0)[r]%
                {$\left(\overline{g^p}-\overline{c^p}\right)^*$}}
                \put(0,55){\dashbox{3}(40,0){}}
                \put(100,-5){\makebox(0, 0)[t]%
                {$\left(\overline{\tau^p}\right)^*$}}
                \put(100,0){\dashbox{3}(0,21){}}
                % Origin strategy
                \thicklines
                \put(-12,-4){\line(0,1){8}}
                %\put(-4,12){\line(1,0){8}}
                \thinlines
                \put(-12,-5){\makebox(0, 0)[t]%
                {$\frac{-1}{\lambda^p}$}}
                %\put(-5,12){\makebox(0, 0)[r]%
                %{$-\frac{c^s}{\lambda^p}$}}
        \end{picture}
        \label{fig:rate_maximization_cycle_visualization}
        }
%        \\
%        \subfloat[Limiting Cases]{
%        \begin{picture}(401.5,115.5)(0,0)
%                \put(0,0){\scalebox{.75}{
%                \begin{picture}(176,154)(-36,-20)
%                        %\put(-36,-20){\framebox(176,154){}}
%                        % Horizontal Axis
%                        \thicklines
%                        \put(-30, 0){\vector(1, 0){144}}
%                        \put(115, 0){\makebox(0, 0)[l]%
%                        {$\overline{\tau^p}$}}
%                        % Vertical Axis
%                        \thicklines
%                        \put(0, -16){\vector(0, 1){136}}
%                        \put(0, 121){\makebox(0, 0)[b]%
%                        {$\overline{g^p}-\overline{c^p}$}}
%                        % Curve, starting origin and moving CW
%                        \thinlines
%                        \qbezier(0,0)(10,55)(40,55)
%                        \qbezier(40,55)(58.5,55)(84,44)
%                        %\qbezier(84,44)(92,41)(114,21)
%                        \qbezier(84,44)(92,41)(100,21)
%                        % Max and mins
%                        \thicklines
%                        \put(100,-4){\line(0,1){8}}
%                        \thinlines
%                        \put(100,-5){\makebox(0, 0)[t]%
%                        {$\left(\overline{\tau^p}\right)^*$}}
%                        \put(100,0){\line(0,1){120}}
%                        % Limits
%                        \put(100, 21){\circle*{10}}
%                        \put(108,60){\rotatebox{90}{\makebox(0,0)[t]%
%                        {\shortstack%
%                        {Time maximizer\\for\\$c^s \to \infty$}}}}
%                        \put(0, 0){\circle*{10}}
%                        \put(-2,60){\rotatebox{90}{\makebox(0,0)[b]%
%                        {\shortstack%
%                        {Time minimizer\\for\\$c^s \to -\infty$}}}}
%                        \put(50,80){\makebox(0,0)%
%                        {$\lim\limits_{c^s\to\pm\infty}\gamma^*%
%                        =\mp \infty$}}
%                \end{picture}        
%                }} % End of \put\scalebox for left figure
%                %
%                \put(134,0){\scalebox{.75}{
%                \begin{picture}(176,154)(-54,-20)
%                        %\put(-54,-20){\framebox(176,154){}}
%                        %
%                \end{picture}
%                }} % End of \put\scalebox for middle figure
%                %
%                \put(268,0){\scalebox{.75}{
%                \begin{picture}(178,154)(-37,-20)
%                        %\put(-37,-20){\framebox(178,154){}}
%                        % Horizontal Axis
%                        \thicklines
%                        \put(-30, 0){\vector(1, 0){144}}
%                        \put(115, 0){\makebox(0, 0)[l]%
%                        {$\overline{\tau^p}$}}
%                        % Vertical Axis
%                        \thicklines
%                        \put(0, -16){\vector(0, 1){136}}
%                        \put(0, 121){\makebox(0, 0)[b]%
%                        {$\overline{g^p}-\overline{c^p}$}}
%                        % Curve, starting origin and moving CW
%                        \thinlines
%                        \qbezier(0,0)(10,55)(40,55)
%                        \qbezier(40,55)(58.5,55)(84,44)
%                        %\qbezier(84,44)(92,41)(114,21)
%                        \qbezier(84,44)(92,41)(100,21)
%                        % Max and mins
%                        \thicklines
%                        \put(100,-4){\line(0,1){8}}
%                        \thinlines
%                        \put(100,-5){\makebox(0, 0)[t]%
%                        {$\left(\overline{\tau}\right)^*$}}
%                        \put(100,0){\line(0,1){120}}
%                        % Limits
%                        \put(100, 21){\circle*{10}}
%                        \put(108,60){\rotatebox{90}{\makebox(0,0)[t]%
%                        {\shortstack%
%                        {Time maximizer\\for\\$G^T \to \infty$}}}}
%                        \put(0, 0){\circle*{10}}
%                        \put(-2,60){\rotatebox{90}{\makebox(0,0)[b]%
%                        {\shortstack%
%                        {Time minimizer\\for\\$G^T \to -\infty$}}}}
%                        \put(50,80){\makebox(0,0)%
%                        {$\lim\limits_{G^T\to\pm\infty}\gamma^*%
%                        =\mp \infty$}}
%                \end{picture}
%                }} % End of \put\scalebox for right figure
%        \end{picture}
%        \label{fig:rate_maximization_cycle_limit_case}
%        }
        \caption[Visualization of Rate Maximization]{Rate maximization.
        Curves show the \aimention{Vilfredo Pareto}Pareto frontiers for
        maximizing net gain mean and maximizing or minimizing total
        time. $G^T$ denotes the net gain survival threshold. $\gamma^*$
        denotes the maximal rate, and the solutions that achieve this
        rate are shown as large dots at points of tangency with a line
        of slope $\gamma^*$. The dotted arrow in
        \subref{fig:rate_maximization_cycle_visualization} shows how the
        figure changes as with $c^s$ and $\lambda^p$. It is not
        necessary that $|\mu^*| < \infty$ and $(\overline{\tau^p} +
        1/\lambda^p)^* < \infty$.}
        \label{fig:rate_maximization}
\end{figure}
%
The curve shows the set of all \aimention{Vilfredo Pareto}Pareto
efficient solutions for maximizing gain and both maximizing or
minimizing time\footnote{That is, points on the curve represent the
highest and lowest expected total times possible for a given expected
net gain.}. In \longref{fig:rate_maximization_cycle_visualization}, the
axes are shifted to highlight the impact of parameter changes on the
solution\footnote{The encounter rate $\lambda^p$ may only be considered
to be an environmental parameter if, for all $i \in \{1,2,\dots,n\}$,
$p_i$ is not free for the agent to choose. The analogous graphical
interpretation of results on classical \ac{OFT} statistics does not have
this constraint.  However, in both cases the shape of the
\aimention{Vilfredo Pareto}Pareto frontier curve will change with
changes in the overall encounter rate if the individual task type
encounter rates are not also scaled by the same factor.}.
\longref{fig:oft_rate_maximization} was generated in an identical way;
however, the single gain curve $g_1(\tau_1)$ is replaced with a curve
representing the \aimention{Vilfredo
Pareto}Pareto-efficient-average-processing-gain curve. This graphical
interpretation of rate maximization applies to any case rather than just
the single type patch case\footnote{Recall that $\overline{g^p}=g_1$ and
$\overline{\tau^p}=\tau_1$ for the $n=1$ (and $p_1=1$) case.}.
Additionally, search and processing costs are shown. In fact, when $G^T
= 0$, it is clear that the cost of searching plays the same role as a
gain success threshold. The line whose slope is the rate may be viewed
as \emph{pivoting} at the point $(-1/\lambda^p,G^T/N^p+c^s/\lambda^p)$,
and so this may be called the \emph{threshold pivot point}. 

For this particular $\overline{g^p}$ and $\overline{\tau^p}$, several
observations can be made.
%
\begin{description}
        \item\emph{Cycle Lifetime and Gain Threshold:} The per-cycle net
                gain threshold $G^T/N^p$ depends upon the number of
                lifetime cycles $N^p$.  Therefore, the length of
                lifetime has an impact on the gain budget and therefore
                the optimal strategy.
        \item\emph{Negative Gain Budgets:} \index{patch
                overstaying!rational explanation}Whenever the expected
                gain is less than the gain success threshold, the
                average time spent processing tasks (which the agent can
                control through both preference probability and
                processing time) will increase. Graphically, whenever
                $G^T/N^p + c^s/\lambda^p >
                (\overline{g^p}-\overline{c^p})^*$, an increase in $G^T$
                will bring an increase in the $\overline{\tau^p}$
                solution. Therefore, rate maximization in classical
                \ac{OFT} may predict low processing times compared to
                observations because costs or required energy thresholds
                are being neglected.
        \item\emph{Sunk Cost Effect and Concorde Fallacy:}
                \index{sunk-cost effect}\index{Concorde
                fallacy|see{sunk-cost effect}}\citet{HA99}
                review both the \emph{sunk cost effect} and the
                \emph{Concorde fallacy}, which are well-known in
                economics and biology. These describe the same behavior,
                except the former is typically applied to humans and the
                latter is typically applied to animals.  Agents are said
                to manifest this behavior when showing a ``greater
                tendency to continue an endeavor once an investment in''
                gain has been made \citep[p.~591]{HA99}. In other
                words, an agent will continue to process tasks evidently
                \emph{because} of the \emph{cost} of prior processing.
                \citeauthor{HA99} suggest that this is due to the use of
                an ordinarily adaptive heuristic rule; however, we
                suggest that rate maximization may also explain this
                effect.  Consider the situation shown in
                \longref{fig:rate_maximization_cycle_visualization}, but
                replace the strictly positive $(\overline{g^p} -
                \overline{c^p})$-$\overline{\tau^p}$ curve with a
                version of itself that has been mirrored around the
                $\tau^p$-axis. In this case, the optimal average
                processing time is $(\tau^p)^*$. That is, it is better
                to face a negative energy budget after a long period of
                time rather than facing a negative energy budget after a
                short period of time, even though the later negative
                energy budget is worse (\ie, it may be better to be
                short in gain after much effort than being short in gain
                after no effort). 
\end{description}
%
Therefore, rate maximization explains some effects that may otherwise be
considered to be \index{rationality|see{ecological
rationality}}\index{ecological rationality}irrational.%
\index{examples!graphical solutions!rate of excess net gain|)indexdef}%
\index{rate of excess net gain|)}%
\index{rate of excess net gain!graphical optimization|)}%

\subsubsection{Efficiency}%
\index{efficiency|see{excess efficiency}}%
\index{excess efficiency|(indexdef}

Now, take an agent that must achieve an expected \emph{gross} gain of at
least \index{gain success threshold}\index{gross gain success
threshold}$G_g^T$ after $N^p$ tasks processed but must achieve its goal
while accumulating as little cost as possible. We consider the objective
function
%
\begin{equation}
        \frac{ \E(G^{N^p}) + \E(C^{N^p}) - G_g^T }{ \E(C^{N^p}) }
\end{equation}
%
or, equivalently,
%
\begin{equation}
        \frac{ \E(G_1) + \E(C_1) - \frac{G_g^T}{N^p} }{ \E(C_1) }
        \label{eq:payoff_efficiency}
\end{equation}
%
So, the goal of the agent is to maximize its expected \emph{excess}
gross gain while also accumulating as little cost as possible. This is a
notion of the agent's \emph{efficiency} (\ie, maximal benefit-to-cost
ratio). While there is no direct pressure on time minimization in this
objective, all costs in the model depend linearly on time.  Therefore,
minimization of cost indirectly has a time minimization effect as well.%
\index{excess efficiency|)indexdef}

\index{excess efficiency|(}%
\index{excess efficiency!graphical optimization|(}%
A \index{examples!graphical solutions!excess
efficiency|(indexdef}graphical solution to this problem is shown in
\longref{fig:efficiency_maximization}.
%
%\begin{figure}[!ht]\centering
\begin{figure}\centering
        \begin{picture}(168,68)(0,0)
                \put(0, 0){\scalebox{0.8}{\framebox(210,85){
                \shortstack[c]%
                {$\E(G_1) + \E(C_1) = \overline{g^p}$\\%
                $\E(C_1) = \overline{c^p} + \frac{c^s}{\lambda^p}$\\%
                $\epsilon \triangleq \frac{\overline{g^p} - %
                \frac{G^T}{N^p}}{\overline{c^p}%
                + \frac{c^s}{\lambda^p}}$\\%
                $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}}
        \end{picture}\\
        \subfloat[Per-Cycle Perspective]{
        \begin{picture}(194,155)(-42,-21)
                % Horizontal Axis
                \thicklines
                %\put(-30, 12){\vector(1, 0){144}}
                \put(-30, 0){\vector(1, 0){144}}
                \put(115, 0){\makebox(0, 0)[l]%
                {$\overline{c^p} + \frac{c^s}{\lambda^p}$}}
                % Vertical Axis
                \thicklines
                \put(-12, -16){\vector(0, 1){136}}
                %\put(0, -16){\vector(0, 1){136}}
                \put(-12, 121){\makebox(0, 0)[b]{$\overline{g^p}$}}
                % Rate Line
                \thinlines
                \put(-12, 92){\line(2,-1){126}}
                \put(84, 44){\circle*{5}}
                % Maximal Rate (Slope)
                \thinlines
                \put(20,76){\line(1,0){34}}
                \put(54,76){\line(0,-1){17}}
                \put(53,75){\makebox(0, 0)[rt]{$\epsilon^*$}}
                % Threshold Value
                \thicklines
                \put(-16, 92){\line(1,0){8}}
                \put(-17, 92){\makebox(0, 0)[r]{$\frac{G_g^T}{N^p}$}}
                % Curve, starting origin and moving CW
                \thinlines
                \qbezier(0,0)(10,55)(40,55)
                \qbezier(40,55)(58.5,55)(84,44)
                %\qbezier(84,44)(92,41)(114,21)
                \qbezier(84,44)(92,41)(100,21)
                % Max and mins
                \thicklines
                \put(-16,55){\line(1,0){8}}
                \put(100,-4){\line(0,1){8}}
                \thinlines
                \put(-17,55){\makebox(0, 0)[r]%
                {$\left(\overline{g^p}\right)^*$}}
                \put(-12,55){\dashbox{3}(52,0){}}
                \put(100,-4){\makebox(0, 0)[t]%
                {$\left(\overline{c^p}%
                +\frac{c^s}{\lambda^p}\right)^*$}}
                \put(100,0){\dashbox{3}(0,21){}}
                % Origin strategy
                \thicklines
                \put(0,-4){\line(0,1){8}}
                \put(-16,0){\line(1,0){8}}
                \thinlines
                \put(0,-5){\makebox(0, 0)[t]%
                {$\frac{c^s}{\lambda^p}$}}
        \end{picture}
        \label{fig:efficiency_maximization_cycle}
        }
        \subfloat[Graphical Interpretation]{
        \begin{picture}(160,155)(-33,-21)
                % Horizontal Axis
                \thicklines
                \put(-30, 0){\vector(1, 0){144}}
                \put(115, 0){\makebox(0, 0)[l]{$\overline{c^p}$}}
                % Vertical Axis
                \thicklines
                \put(0, -16){\vector(0, 1){136}}
                \put(0, 121){\makebox(0, 0)[b]{$\overline{g^p}$}}
                % Rate Line
                \thinlines
                \put(-12, 92){\line(2,-1){126}}
                \put(84, 44){\circle*{5}}
                % Maximal Rate (Slope)
                \thinlines
                \put(20,76){\line(1,0){34}}
                \put(54,76){\line(0,-1){17}}
                \put(53,75){\makebox(0, 0)[rt]{$\epsilon^*$}}
                % Threshold Value
                \thicklines
                \put(-4, 92){\line(1,0){8}}
                \put(5, 90){\makebox(0, 0)[lb]%
                {$\frac{G_g^T}{N^p}$}}
                \put(0, 92){\circle*{2}}
                \put(-2, 92){\circle*{1}}
                \put(-4, 92){\circle*{1}}
                \put(-6, 92){\circle*{1}}
                \put(-8, 92){\circle*{1}}
                \put(-10, 92){\circle*{1}}
                \put(-12, 92){\circle*{3}}
                \put(-14, 92){\circle*{1}}
                \put(-16, 92){\circle*{1}}
                \put(-18, 92){\circle*{1}}
                \put(-20, 92){\circle*{1}}
                \put(-22, 92){\circle*{1}}
                \put(-24, 92){\circle*{1}}
                \put(-26, 92){\circle*{1}}
                \put(-28, 92){\circle*{1}}
                \put(-30, 92){\vector(-1,0){1}}
                % Curve, starting origin and moving CW
                \thinlines
                \qbezier(0,0)(10,55)(40,55)
                \qbezier(40,55)(58.5,55)(84,44)
                %\qbezier(84,44)(92,41)(114,21)
                \qbezier(84,44)(92,41)(100,21)
                % Max and mins
                \thicklines
                \put(-4,55){\line(1,0){8}}
                \put(100,-4){\line(0,1){8}}
                \thinlines
                \put(-5,54){\makebox(0, 0)[r]%
                {$\left(\overline{g^p}\right)^*$}}
                \put(0,55){\dashbox{3}(40,0){}}
                \put(100,-5){\makebox(0, 0)[t]%
                {$\left(\overline{c^p}\right)^*$}}
                \put(100,0){\dashbox{3}(0,21){}}
                % Origin strategy
                \thicklines
                \put(-12,-4){\line(0,1){8}}
                %\put(-4,12){\line(1,0){8}}
                \thinlines
                \put(-12,-5){\makebox(0, 0)[t]%
                {$\frac{-c^s}{\lambda^p}$}}
                %\put(-5,12){\makebox(0, 0)[r]%
                %{$-\frac{c^s}{\lambda^p}$}}
        \end{picture}
        \label{fig:efficiency_maximization_cycle_visualization}
        }
        \caption[Visualization of Efficiency Maximization]{Efficiency
        maximization. Curves show \aimention{Vilfredo Pareto}Pareto
        frontiers for maximizing gross gain mean and minimizing total
        cost. $G_g^T$ denotes the gross gain survival threshold.
        $\epsilon^*$ denotes the maximal efficiency, and the solutions
        that achieve this efficiency are shown as large dots at points
        of tangency with a line of slope $\epsilon^*$. The dotted arrow
        in \subref{fig:efficiency_maximization_cycle_visualization}
        shows how the figure changes as with $c^s$ and $\lambda^p$. It
        is not necessary that $|(\overline{g^p})^*| < \infty$ and
        $|(\overline{c^p} + c^s/\lambda^p)^*| < \infty$.}
        \label{fig:efficiency_maximization}
\end{figure}
%
Here, the curve shows the set of all \aimention{Vilfredo Pareto}Pareto
efficient solutions for maximizing gross gain and both maximizing or
minimizing cost\footnote{That is, points on the curve represent the
highest and lowest expected cost possible for a given expected gross
gain.}. In \longref{fig:efficiency_maximization_cycle_visualization},
the axes are shifted to highlight the impact of parameter changes on the
solution\footnote{As with the graphical interpretation of rate
maximization, the effects of parameter changes are simpler to explore
using classical \ac{OFT} statistics because the overall encounter rate
is not influenced by the preference probabilities, which are often
considered to be decision variables that describe agent behavior.}. The
line whose slope is the efficiency may be viewed as \emph{pivoting} at
the point $(-c^s/\lambda^p,G_g^T/N^p)$, and so this may be called the
\emph{threshold pivot point}.

For this particular $\overline{g^p}$ and $\overline{c^p}$, several
observations can be made.
%
\begin{description}
        \item\emph{Cycle Lifetime and Gain Threshold:} As with rate
                maximization, the per-cycle gross gain threshold
                $G_g^T/N^p$ depends upon the number of lifetime cycles
                $N^p$. Therefore, the length of lifetime has an impact
                on the gain budget and the optimal strategy.
        \item\emph{Negative Gain Budgets:} Assuming the gain threshold
                is high, the optimal behavior moves toward has a higher
                cost and a lesser gross gain as the gross gain threshold
                is increased. In other words, the number of points
                \emph{lost} per unit cost (\ie, the loss efficiency) is
                actually lower at a higher cost. 
        \item\emph{Negative Search Costs:} Assume that $c^s < 0$ and
                $G_g^T \leq 0$. In this case, the optimal solution is
                the one such that $-c^s/\lambda = \overline{c^p}$. In
                other words, when facing a positive energy budget and a
                search \emph{gain}, the most efficient solution is the
                one where search gains are equal to processing costs. A
                similar result holds for negative average search costs.
                Essentially, whenever the gain threshold pivot point is
                below (or encircled by) the \aimention{Vilfredo
                Pareto}Pareto frontiers, the optimal solution will be
                the one that makes $\E(C_1)=0$.
        \item\emph{Sunk Cost Effect and Concorde Fallacy:}
                \index{sunk-cost effect}Efficiency
                maximization provides an explanation for the sunk cost
                effect and the Concorde fallacy similar to the
                explanation for rate maximization. If the
                $\overline{g^p}$-$\overline{c^p}$ curve in
                \longref{fig:efficiency_maximization_cycle_visualization}
                is mirrored about the $\overline{c^p}$-axis, the
                resulting optimal cost will be $(\overline{c^p})^*$.
                That is, processing cost is maximized in this case.
                Roughly, it is ``better'' to have a negative gain budget
                at high cost than low cost.
\end{description}
%
As with rate maximization, an analysis of this optimization objective
suggests explanations for behaviors that might normally be considered
\index{ecological rationality}irrational.%
\index{examples!graphical solutions!excess efficiency|)indexdef}%
\index{excess efficiency!graphical optimization|)}%
\index{excess efficiency|)}%

\subsubsection{Risk-Sensitivity: Reward-to-Variability Ratios}%
\index{reward-to-variability ratio|(}

As discussed, \citet{Sharpe66} introduces an index called the
\emph{reward-to-variability ratio}\footnote{Again, this is often called
the \emph{\citeauthor{Sharpe66} ratio} by those other than
\citeauthor{Sharpe66}.}, a measure of future performance that balances
expected return with standard deviation of return. When returns are
location-scale distributed with identical skewness for each
location-scale choice (\eg, normally distributed), maximization of this
index is identical to minimization of risk of return shortfall.%
\index{reward-to-variability ratio|)}

\index{reward-to-variability ratio|(indexdef}% 
Let \index{net gain success threshold|see{gain success
threshold}}\index{gain success threshold}$G^T$ represent a
(deterministic) net gain threshold of success for the $N^p$ cycles in an
agent's lifetime. In our context\footnote{For simplicity, we assume
\longrefs{eq:cycle_proc_gross_gain_2moment}--%
\shortref{eq:cycle_net_gain_2moment}.}, the reward-to-variability ratio
is expressed by
%
\begin{equation*}
        \frac{ \E(G^{N^p}) - G^T }{ \std( G^{N^p} ) }
        \quad \text{ or, equivalently, } \quad
        \sqrt{N^p} \frac{ \E(G_1) - \frac{G^T}{N^p} }{ \std( G_1 ) }
\end{equation*}
%
That is, this is a ratio of \emph{excess} returns to the standard
deviation of those returns.%
\index{reward-to-variability ratio|)indexdef}% 

\index{reward-to-variability ratio|(}% 
\index{reward-to-variability ratio!graphical optimization|(}% 
A \index{examples!graphical solutions!reward-to-variability
ratio|(indexdef} graphical solution to this problem is shown in
\longref{fig:risk_sensitivity_sharpe}.
%
%\begin{figure}[!ht]\centering
\begin{figure}\centering
        \begin{picture}(168,68)(0,0)
                \put(0, 0){\scalebox{0.8}{\framebox(210,85){
                \shortstack[c]%
                {$\mu \triangleq \E(G_1) = %
                \overline{g^p} - \overline{c^p} - \frac{c^s}{\lambda^p}%
                $\\%
                $\sigma \triangleq \std(G_1) = \sqrt{%
                \var\left(g^p - c^p\right) %
                + \left(\frac{c^s}{\lambda^p}\right)^2%
                }$\\%
                $\rho \triangleq \frac{\mu - %
                \frac{G^T}{N^p}}{\sigma}$\\%
                $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}}
        \end{picture}\\
        \subfloat[Lifetime Perspective]{
        \begin{picture}(214,149)(-30,-22)
                % Horizontal Axis
                \thicklines
                \put(-10, 12){\vector(1, 0){145}}
                \put(137, 12){\makebox(0, 0)[l]{$\std(G^{N^p})$}}
                % Vertical Axis
                \thicklines
                \put(12, -15){\vector(0, 1){125}}
                \put(12, 111){\makebox(0, 0)[b]{$\E(G^{N^p})$}}
                % Reward-to-Variability Line
                \thinlines
                \put(12, 94){\line(2,-1){123}}
                \put(100, 50){\circle*{5}}
                % Maximal Reward-to-Variability (Slope)
                \thinlines
                \put(16,92){\line(0,-1){29}}
                \put(16,63){\line(1,0){58}}
                \put(16,64){\makebox(0, 0)[lb]{$\sqrt{N^p} \rho^*$}}
                % Threshold Value
                \thicklines
                \put(8, 94){\line(1,0){8}}
                \put(7, 94){\makebox(0, 0)[r]{$G^T$}}
                % Curve, starting from upper left and moving CW
                \thinlines
                \qbezier(35,35)(35,60)(51,60)
                \qbezier(51,60)(80,60)(100,50)
                \qbezier(100,50)(120,40)(120,33)
                \qbezier(120,33)(120,10)(24,-5)
                % Max and mins
                \thicklines
                \put(120,8){\line(0,1){8}}
                \put(8,60){\line(1,0){8}}
                \thinlines
                \put(120,7){\makebox(0, 0)[t]{$\sqrt{N^p} \sigma^*$}}
                \put(120,12){\dashbox{3}(0,21){}}
                \put(7,60){\makebox(0, 0)[r]{$N^p \mu^*$}}
                \put(12,60){\dashbox{3}(39,0){}}
                % Origin strategy
                \thicklines
                \put(24,8){\line(0,1){8}}
                \put(8,-5){\line(1,0){8}}
                \thinlines
                \put(16,17){\makebox(0, 0)[lb]%
                {$\sqrt{N^p} \frac{|c^s|}{\lambda^p}$}}
                %\put(24,-5){\dashbox{3}(0,17){}}
                \put(7,-3){\makebox(0, 0)[r]%
                {$-N^p \frac{c^s}{\lambda^p}$}}
                %\put(12,-5){\dashbox{3}(12,0){}}
        \end{picture}
        \label{fig:risk_sensitivity_sharpe_lifetime}
        }
        \subfloat[Per-Cycle Perspective]{
        \begin{picture}(160,149)(-14,-22)
                \thicklines
                % Horizontal Axis
                \put(-10, 12){\vector(1, 0){145}}
                %\put(137, 12){\makebox(0, 0)[l]{$\std(G_1)$}}
                \put(137, 12){\makebox(0, 0)[l]{$\sigma$}}
                % Vertical Axis
                \put(12, -15){\vector(0, 1){125}}
                %\put(12, 111){\makebox(0, 0)[b]{$\E(G_1)$}}
                \put(12, 111){\makebox(0, 0)[b]{$\mu$}}
                % Reward-to-Variability Line
                \thinlines
                \put(12, 85.2){\line(5,-2){123}}
                \put(110, 46){\circle*{5}}
                % Maximal Reward-to-Variability (Slope)
                \thinlines
                \put(20,82){\line(0,-1){20}}
                \put(20,62){\line(1,0){50}}
                \put(22,64){\makebox(0, 0)[lb]{$\rho^*$}}
                % Threshold Value
                \thicklines
                \put(8,85.2){\line(1,0){8}}
                \put(7,85.2){\makebox(0,0)[r]{$\frac{G^T}{N^p}$}}
                % Curve, starting from upper left and moving CW
                \thinlines
                \qbezier(35,30)(35,55)(51,55)
                \qbezier(51,55)(80,55)(110,46)
                \qbezier(110,46)(120,40)(120,33)
                \qbezier(120,33)(120,10)(24,0)
                % Max and mins
                \thicklines
                \put(120,8){\line(0,1){8}}
                \put(8,55){\line(1,0){8}}
                \thinlines
                \put(120,7){\makebox(0, 0)[t]{$\sigma^*$}}
                \put(120,12){\dashbox{3}(0,21){}}
                \put(7,55){\makebox(0, 0)[r]{$\mu^*$}}
                \put(12,55){\dashbox{3}(39,0){}}
                % Origin strategy
                \thicklines
                \put(24,8){\line(0,1){8}}
                \put(8,0){\line(1,0){8}}
                \thinlines
                \put(24,17){\makebox(0, 0)[b]{$\frac{|c^s|}{\lambda^p}$}}
                %\put(24,0){\dashbox{3}(0,12){}}
                \put(7,0){\makebox(0, 0)[r]{$-\frac{c^s}{\lambda^p}$}}
                %\put(12,0){\dashbox{3}(12,0){}}
        \end{picture}
        \label{fig:risk_sensitivity_sharpe_cycle}
        }
        \\
        \subfloat[Graphical Interpretation]{
        \begin{picture}(236,149)(-52,-22)
                \thicklines
                % Horizontal Axis
                \put(-10, 0){\vector(1, 0){155}}
                \put(147, 0){\makebox(0, 0)[l]%
                {$\sigma-\frac{|c^s|}{\lambda^p}$}}
                % Vertical Axis
                \put(24, -15){\vector(0, 1){125}}
                \put(24, 111){\makebox(0, 0)[b]%
                %{$\mu + \frac{c^s}{\lambda}$}}
                {$\overline{g^p} - \overline{c^p}$}}
                % Reward-to-Variability Line
                \thinlines
                \put(12, 85.2){\line(5,-2){133}}
                \put(110, 46){\circle*{5}}
                % Maximal Reward-to-Variability (Slope)
                \thinlines
                \put(31,77.6){\line(1,0){45}}
                \put(76,77.6){\line(0,-1){18}}
                \put(75,76.6){\makebox(0, 0)[rt]{$\rho^*$}}
                % Threshold Value
                %\thinlines
                %\put(0,85.2){\dashbox{3}(12,0){}}
                \thicklines
                \put(20,73.2){\line(1,0){8}}
                \put(12,-4){\line(0,1){8}}
                \thinlines
                %\put(-3,95.2){\makebox(0,0)[r]{$-\frac{c^s}{|c^s|}$}}
                \put(-3,95.2){\makebox(0,0)[r]{$-\sgn(c^s)$}}
                \put(-2,99.2){\line(0,-1){8}}
                \put(-2,91.2){\line(1,0){8}}
                \put(24,73.2){\circle*{1}}
                \put(22,75.2){\circle*{1}}
                \put(20,77.2){\circle*{1}}
                \put(18,79.2){\circle*{1}}
                \put(16,81.2){\circle*{1}}
                \put(14,83.2){\circle*{1}}
                \put(12,85.2){\circle*{3}}
                \put(10,87.2){\circle*{1}}
                \put(8,89.2){\circle*{1}}
                \put(6,91.2){\circle*{1}}
                \put(4,93.2){\circle*{1}}
                \put(2,95.2){\circle*{1}}
                \put(0,97.2){\circle*{1}}
                \put(-2,99.2){\circle*{1}}
                \put(-4,101.2){\circle*{1}}
                \put(-6,103.2){\circle*{1}}
                \put(-8,105.2){\circle*{1}}
                \put(-10,107.2){\circle*{1}}
                \put(-12,109.2){\vector(-1, 1){1}}
                \put(17,73.2){\makebox(0,0)[r]{$\frac{G^T}{N^p}$}}
                \put(12,-5){\makebox(0,0)[t]%
                {$-\frac{|c^s|}{\lambda^p}$}}
                % Curve, starting from upper left and moving CW
                \thinlines
                \qbezier(35,30)(35,55)(51,55)
                \qbezier(51,55)(80,55)(110,46)
                \qbezier(110,46)(120,40)(120,33)
                \qbezier(120,33)(120,10)(24,0)
                % Max and mins
                \thicklines
                \put(120,-4){\line(0,1){8}}
                \put(20,55){\line(1,0){8}}
                \thinlines
                \put(120,-5){\makebox(0, 0)[t]%
                {$\sigma^*-\frac{|c^s|}{\lambda^p}$}}
                \put(120,0){\dashbox{3}(0,33){}}
                \put(129,33){\circle*{1}}
                \put(126,33){\circle*{1}}
                \put(123,33){\circle*{1}}
                \put(120,33){\circle*{1}}
                \put(117,33){\circle*{1}}
                \put(114,33){\circle*{1}}
                \put(111,33){\circle*{1}}
                \put(108,33){\circle*{1}}
                \put(105,33){\circle*{1}}
                \put(102,33){\vector(-1, 0){1}}
                \put(19,55){\makebox(0, 0)[r]%
                {$\left( \overline{g^p} - \overline{c^p} \right)^*$}}
                \put(24,55){\dashbox{3}(27,0){}}
        \end{picture}
        \label{fig:risk_sensitivity_sharpe_cycle_visualization}
        }
%        \subfloat[Limiting Cases]{
%        \begin{picture}(111,150)(-4,0)
%                \put(0,75){\scalebox{.5}{
%                \begin{picture}(215,149)(-31,-22)
%                        \thicklines
%                        % Horizontal Axis
%                        \put(-10, 0){\vector(1, 0){155}}
%                        \put(147, 0){\makebox(0, 0)[l]%
%                        {$\sigma-\frac{|c^s|}{\lambda^p}$}}
%                        % Vertical Axis
%                        \put(24, -15){\vector(0, 1){125}}
%                        \put(24, 111){\makebox(0, 0)[b]%
%                        {$\overline{g^p} - \overline{c^p}$}}
%                        % Reward-to-Variability Line
%                        \thinlines
%                        \put(-10, 79){\line(1,-1){94}}
%                        \put(-10, 11){\line(1,1){99}}
%                        \put(24, 45){\circle*{10}}
%                        % Threshold Value
%                        \thicklines
%                        %\put(20,73.2){\line(1,0){8}}
%                        \thinlines
%                        \put(-6,57)%
%                        {\scalebox{0.75}{\makebox(0,0)[lb]{$-1$}}}
%                        \put(-7,76){\line(0,-1){20}}
%                        \put(-7,56){\line(1,0){20}}
%                        \put(0,32)%
%                        {\scalebox{0.75}{\makebox(0,0)[lt]{$1$}}}
%                        \put(-1,20){\line(0,1){14}}
%                        \put(-1,34){\line(1,0){14}}
%                        %\put(21,73.2){\scalebox{0.75}{%
%                        %\makebox(0,0)[rb]{$\frac{G^T}{N^p}$}}}
%                        \linethickness{3.0\unitlength}
%                        \put(24,55){\line(0,-1){55}}
%                        % Max and mins
%                        %\thicklines
%                        %\put(20,55){\line(1,0){8}}
%                        \thinlines
%                        %\put(19,55){\makebox(0, 0)[r]%
%                        %{$\left(\overline{g^p}%
%                        %-\overline{c^p}\right)^*$}}
%                        \put(34,57){\rotatebox{45}{
%                        \makebox(0,0)[lb]{$c^s < 0$}}}
%                        \put(34,33){\rotatebox{315}{
%                        \makebox(0,0)[lt]{$c^s > 0$}}}
%                        \put(84,52.5)%
%                        {\makebox(100,15)
%                        {\shortstack{Optimal Anywhere\\for\\%
%                        $\frac{|c^s|}{\lambda^p} \to
%                        \infty$\\[17\unitlength]%
%                        $\rho^* = -\sgn(c^s)$}}}
%                \end{picture}}
%                } % End of \put\scalebox for upper figure
%                %
%                \put(0,0){\scalebox{.5}{
%                \begin{picture}(215,149)(-31,-22)
%                        \thicklines
%                        % Horizontal Axis
%                        \put(-10, 0){\vector(1, 0){155}}
%                        \put(147, 0){\makebox(0, 0)[l]%
%                        {$\sigma-\frac{|c^s|}{\lambda^p}$}}
%                        % Vertical Axis
%                        \put(24, -15){\vector(0, 1){125}}
%                        \put(24, 111){\makebox(0, 0)[b]%
%                        {$\overline{g^p} - \overline{c^p}$}}
%                        % Reward-to-Variability Line
%                        \thinlines
%                        \put(120, 0){\line(0,1){110}}
%                        \put(126,52.5){\rotatebox{90}{\makebox(0,0)[t]%
%                        {\shortstack%
%                        {Scale maximizer\\for\\$G^T \to \infty$}}}}
%                        \put(22,52.5){\rotatebox{90}{\makebox(0,0)[b]%
%                        {\shortstack%
%                        {Scale minimizer\\for\\$G^T \to -\infty$}}}}
%                        \put(120, 33){\circle*{10}}
%                        \put(24, 0){\circle*{10}}
%                        \put(72,80){\makebox(0,0)%
%                        {$\lim\limits_{G^T\to\pm\infty} \rho^*%
%                        =\mp \infty$}}
%                        % Threshold Value
%                        \thinlines
%                        % Curve, starting from upper left and moving CW
%                        \thinlines
%                        \qbezier(35,30)(35,55)(51,55)
%                        \qbezier(51,55)(80,55)(110,46)
%                        \qbezier(110,46)(120,40)(120,33)
%                        \qbezier(120,33)(120,10)(24,0)
%                        % Max and mins
%                        \thicklines
%                        \put(120,-4){\line(0,1){8}}
%                        \thinlines
%                        \put(120,-5){\makebox(0, 0)[t]%
%                        {$\sigma^*-\frac{|c^s|}{\lambda^p}$}}
%                \end{picture}} % End of \put\scalebox for lower figure
%        }
%        \end{picture}
%        \label{fig:risk_sensitivity_sharpe_cycle_limit_case}
%        }
        \caption[Visualization of Reward-to-Variability
        Maximization]{Reward-to-variability maximization. Curves show
        \aimention{Vilfredo Pareto}Pareto frontiers for maximizing mean
        net gain and minimizing net gain scale. $G^T$ denotes the net
        gain survival threshold.  $\rho^*$ denotes the maximal per-cycle
        reward-to-variability ratio, and the solutions that achieve this
        ratio are shown as large dots at points of tangency with a line
        of slope $\rho^*$.  The dotted arrows in
        \subref{fig:risk_sensitivity_sharpe_cycle_visualization} show
        how the figure changes as $|c^s|/\lambda^p$ increases. It is not
        necessary that $|\mu^*| < \infty$ and $\sigma^* < \infty$.}
        \label{fig:risk_sensitivity_sharpe}
\end{figure}
%
The curve represents the \aimention{Vilfredo Pareto}Pareto frontiers for
maximizing expected net gain and minimizing or maximizing net gain
standard deviation. Notice that the shape of the curve shown in
\longref{fig:risk_sensitivity_sharpe_lifetime} differs slightly from the
shape of the curve in \longref{fig:risk_sensitivity_sharpe_cycle} due to
different scaling of the axes. Even when $\lambda^p$ may be considered a
parameter of the environment (\ie, not a function of decision
variables), the shape of the \aimention{Vilfredo Pareto}Pareto frontier
curve is still dependent upon it. Therefore, the graphical
interpretation of maximization in
\longref{fig:risk_sensitivity_sharpe_cycle_visualization} requires
horizontally squeezing or stretching the curve with changes in
$\lambda^p$.

For this particular $\overline{g^p}$ and $\overline{c^p}$, several
observations can be made.
%
\begin{description}
        \item\emph{Cycle Lifetime and Gain Threshold:} Similarly as with
                rate and efficiency maximization, the per-cycle gross
                gain threshold $G^T/N^p$ depends upon the number of
                lifetime cycles $N^p$. Therefore, the length of lifetime
                has an impact on the gain budget and therefore the
                optimal strategy.
        \item\emph{Negative Gain Budgets:} Assuming the gain threshold
                is high, the optimal behavior moves toward has a higher
                standard deviation and a lesser net gain as the net gain
                threshold is increased. In other words, the number of
                points \emph{lost} per unit standard deviation is
                actually lower at a higher standard deviation. Put
                another way, additional uncertainty in rewards is less
                costly because it comes with a higher probability that
                returns will be above the gain threshold.
\end{description}
%
As with rate and efficiency maximization, an analysis of this
optimization objective suggests explanations for behaviors that might
normally be considered \index{ecological rationality}irrational.%
\index{examples!graphical solutions!reward-to-variability
ratio|)indexdef}%
\index{reward-to-variability ratio!graphical optimization|)}% 
\index{reward-to-variability ratio|)}% 

\subsubsection{Reward-to-Variance Ratios}%
\index{reward-to-variance ratio|(}% 

Optimization of portfolio performance based entirely on expectation and
variance was originally developed by \citet{Markowitz52} and
\citet{Tobin58}. \Citet{Sharpe66} makes the natural substitution of
standard deviation for variance in the definition of the
reward-to-variability ratio. While this substitution allows maximization
of the performance index to have a risk minimization interpretation, it
makes analytical derivation of the optimal behavior difficult.  To
compensate, we introduce a \emph{reward-to-variance} ratio that is
motivated by the reward-to-variability ratio but provides greater
analytical tractability.\index{reward-to-variance ratio|)}% 

\index{reward-to-variance ratio|(indexdef}% 
Let \index{gain success threshold}$G^T$ represent a (deterministic) net
gain threshold of success for the $N^p$ cycles in an agent's lifetime.
In our context\footnote{For simplicity, we assume
\longrefs{eq:cycle_proc_gross_gain_2moment}--%
\shortref{eq:cycle_net_gain_2moment}.}, the reward-to-variance ratio is
expressed by
%
\begin{equation*}
        \frac{ \E(G^{N^p}) - G^T }{ \var( G^{N^p} ) }
        \quad \text{ or, equivalently, } \quad
        N^p \frac{ \E(G_1) - \frac{G^T}{N^p} }{ \var( G_1 ) }
\end{equation*}
%
That is, this is a ratio of \emph{excess} returns to the \emph{variance}
of those returns.%
\index{reward-to-variance ratio|)indexdef}% 

\index{reward-to-variance ratio|(}% 
\index{reward-to-variance ratio!graphical optimization|(}% 
A \index{examples!graphical solutions!reward-to-variance
ratio|(indexdef}graphical solution to this problem is shown in
\longref{fig:risk_sensitivity_variance}.
%
%\begin{figure}[!ht]\centering
\begin{figure}\centering
        \begin{picture}(168,68)(0,0)
                \put(0, 0){\scalebox{0.8}{\framebox(210,85){
                \shortstack[c]%
                {$\mu \triangleq \E(G_1) = %
                \overline{g^p} - \overline{c^p} - \frac{c^s}{\lambda^p}%
                $\\%
                $\sigma^2 \triangleq \var(G_1) = %
                \var\left(g^p - c^p\right) %
                + \left(\frac{c^s}{\lambda^p}\right)^2$%
                \\%
                $\eta \triangleq \frac{\mu - %
                \frac{G^T}{N^p}}{\sigma^2}$\\%
                $\lozenge^* \triangleq \max\left\{\lozenge\right\}$}}}}
        \end{picture}\\
        \subfloat[Lifetime Perspective]{
        \begin{picture}(214,149)(-30,-22)
                % Horizontal Axis
                \thicklines
                \put(-10, 12){\vector(1, 0){145}}
                \put(137, 12){\makebox(0, 0)[l]{$\var(G^{N^p})$}}
                % Vertical Axis
                \thicklines
                \put(12, -15){\vector(0, 1){125}}
                \put(12, 111){\makebox(0, 0)[b]{$\E(G^{N^p})$}}
                % Reward-to-Variance Line
                \thinlines
                \put(12, 94){\line(2,-1){123}}
                \put(100, 50){\circle*{5}}
                % Maximal Reward-to-Variance (Slope)
                \thinlines
                \put(30,85){\line(1,0){34}}
                \put(64,85){\line(0,-1){17}}
                \put(63,84){\makebox(0, 0)[rt]{$\eta^*$}}
                % Threshold Value
                \thicklines
                \put(8, 94){\line(1,0){8}}
                \put(7, 94){\makebox(0, 0)[r]{$G^T$}}
                % Curve, starting from upper left and moving CW
                \thinlines
                \qbezier(35,35)(35,60)(51,60)
                \qbezier(51,60)(80,60)(100,50)
                \qbezier(100,50)(120,40)(120,33)
                \qbezier(120,33)(120,10)(29,0)
                % Max and mins
                \thicklines
                \put(120,8){\line(0,1){8}}
                \put(8,60){\line(1,0){8}}
                \thinlines
                \put(120,7){\makebox(0, 0)[t]{$N^p (\sigma^2)^*$}}
                \put(120,12){\dashbox{3}(0,21){}}
                \put(7,60){\makebox(0, 0)[r]{$N^p \mu^*$}}
                \put(12,60){\dashbox{3}(39,0){}}
                % Origin strategy
                \thicklines
                \put(29,8){\line(0,1){8}}
                \put(8,0){\line(1,0){8}}
                \thinlines
                \put(25,17){\makebox(0, 0)[lb]%
                {$N^p \left(\frac{c^s}{\lambda^p}\right)^2$}}
                \put(7,-3){\makebox(0, 0)[r]%
                {$-N^p \frac{c^s}{\lambda^p}$}}
        \end{picture}
        \label{fig:risk_sensitivity_variance_lifetime}
        }
        \subfloat[Per-Cycle Perspective]{
        \begin{picture}(163,149)(-14,-22)
                \thicklines
                % Horizontal Axis
                \put(-10, 12){\vector(1, 0){145}}
                \put(137, 12){\makebox(0, 0)[l]{$\sigma^2$}}
                % Vertical Axis
                \put(12, -15){\vector(0, 1){125}}
                \put(12, 111){\makebox(0, 0)[b]{$\mu$}}
                % Reward-to-Variance Line
                \thinlines
                \put(12, 94){\line(2,-1){123}}
                \put(100, 50){\circle*{5}}
                % Maximal Reward-to-Variance (Slope)
                \thinlines
                \put(30,85){\line(1,0){34}}
                \put(64,85){\line(0,-1){17}}
                \put(63,84){\makebox(0, 0)[rt]{$\eta^*$}}
                % Threshold Value
                \thicklines
                \put(8, 94){\line(1,0){8}}
                \put(7, 94){\makebox(0, 0)[r]{$\frac{G^T}{N^p}$}}
                % Curve, starting from upper left and moving CW
                \thinlines
                \qbezier(35,35)(35,60)(51,60)
                \qbezier(51,60)(80,60)(100,50)
                \qbezier(100,50)(120,40)(120,33)
                \qbezier(120,33)(120,10)(29,0)
                % Max and mins
                \thicklines
                \put(120,8){\line(0,1){8}}
                \put(8,60){\line(1,0){8}}
                \thinlines
                \put(120,7){\makebox(0, 0)[t]{$(\sigma^2)^*$}}
                \put(120,12){\dashbox{3}(0,21){}}
                \put(7,60){\makebox(0, 0)[r]{$\mu^*$}}
                \put(12,60){\dashbox{3}(39,0){}}
                % Origin strategy
                \thicklines
                \put(29,8){\line(0,1){8}}
                \put(8,0){\line(1,0){8}}
                \thinlines
                \put(29,17){\makebox(0, 0)[b]%
                {$\left(\frac{c^s}{\lambda^p}\right)^2$}}
                \put(7,-3){\makebox(0, 0)[r]%
                {$-\frac{c^s}{\lambda^p}$}}
        \end{picture}
        \label{fig:risk_sensitivity_variance_cycle}
        }
        \\
        \subfloat[Graphical Interpretation]{
        \begin{picture}(239,149)(-30,-23)
                \thicklines
                % Horizontal Axis
                \put(-10, 0){\vector(1, 0){155}}
                \put(147, 0){\makebox(0, 0)[l]%
                {$\var\left(g^p - c^p\right)$}}
                % Vertical Axis
                \put(24, -15){\vector(0, 1){125}}
                \put(24, 111){\makebox(0, 0)[b]%
                {$\overline{g^p} - \overline{c^p}$}}
                % Reward-to-Variance Line
                \thinlines
                \put(7, 96.5){\line(2,-1){128}}
                \put(100, 50){\circle*{5}}
                % Maximal Reward-to-Variance (Slope)
                \thinlines
                \put(30,85){\line(0,-1){17}}
                \put(30,68){\line(1,0){34}}
                \put(31,70){\makebox(0, 0)[lb]{$\eta^*$}}
                % Threshold Value
                \thicklines
                \put(20,96.5){\line(1,0){8}}
                %\put(20,84.5){\line(1,0){8}}
                \put(7,-4){\line(0,1){8}}
                \thinlines
                \put(23,87.41){\circle*{1}}
                \put(22,88.62){\circle*{1}}
                \put(21,89.54){\circle*{1}}
                \put(20,90.32){\circle*{1}}
                \put(19,91){\circle*{1}}
                \put(17,92.2){\circle*{1}}
                \put(15,93.23){\circle*{1}}
                \put(13,94.15){\circle*{1}}
                \put(11,94.99){\circle*{1}}
                \put(9,95.77){\circle*{1}}
                \put(7,96.5){\circle*{3}}
                \put(5,97.19){\circle*{1}}
                \put(3,97.84){\circle*{1}}
                \put(1,98.46){\circle*{1}}
                \put(-1,99.05){\circle*{1}}
                \put(-3,99.62){\circle*{1}}
                \put(-5,100.17){\circle*{1}}
                \put(-7,100.71){\circle*{1}}
                \put(-9,101.22){\circle*{1}}
                \put(-11,101.72){\circle*{1}}
                \put(-13,102.20){\vector(-5, 1){1}}
                %\put(17, 84.5){\makebox(0, 0)[r]{$\frac{G^T}{N^p}$}}
                \put(29, 96.5){\makebox(0, 0)[l]%
                {$\frac{G^T}{N^p}+\frac{c^s}{\lambda^p}$}}
                \put(4,-5){\makebox(0,0)[t]%
                {$-\left(\frac{c^s}{\lambda^p}\right)^2$}}
                % Curve, starting from upper left and moving CW
                \thinlines
                \qbezier(35,35)(35,60)(51,60)
                \qbezier(51,60)(80,60)(100,50)
                \qbezier(100,50)(120,40)(120,33)
                \qbezier(120,33)(120,10)(29,0)
                % Max and mins
                \thicklines
                \put(120,-4){\line(0,1){8}}
                \put(20,60){\line(1,0){8}}
                \thinlines
                \put(120,-5){\makebox(0, 0)[t]%
                {$\left(\var\left(g^p-c^p\right)\right)^*$}}
                \put(120,0){\dashbox{3}(0,33){}}
                \put(19,60){\makebox(0, 0)[r]%
                {$\left( \overline{g^p} - \overline{c^p} \right)^*$}}
                \put(24,60){\dashbox{3}(27,0){}}
        \end{picture}
        \label{fig:risk_sensitivity_variance_cycle_visualization}
        }
%        \subfloat[Limiting Cases]{
%        \begin{picture}(111,150)(-4,0)
%                \put(0,75){\scalebox{.5}{
%                \begin{picture}(241,149)(-31,-22)
%                        \thicklines
%                        % Horizontal Axis
%                        \put(-10, 0){\vector(1, 0){155}}
%                        \put(147, 0){\makebox(0, 0)[l]%
%                        {$\var\left(g^p-c^p\right)$}}
%                        % Vertical Axis
%                        \put(24, -15){\vector(0, 1){125}}
%                        \put(24, 111){\makebox(0, 0)[b]%
%                        {$\overline{g^p} - \overline{c^p}$}}
%                        % Reward-to-Variance Line
%                        \thinlines
%                        \put(-10, 50){\line(1,0){155}}
%                        \put(100, 50){\circle*{10}}
%                        % Maximal Reward-to-Variance (Slope)
%                        \thinlines
%                        % Threshold Value
%                        %\thicklines
%                        %\put(20,84.5){\line(1,0){8}}
%                        \thinlines
%                        %\put(17,84.5){\scalebox{0.75}{%
%                        %\makebox(0,0)[rb]{$\frac{G^T}{N^p}$}}}
%                        % The curve
%                        \thinlines
%                        \qbezier(35,35)(35,60)(51,60)
%                        \qbezier(51,60)(80,60)(100,50)
%                        \qbezier(100,50)(120,40)(120,33)
%                        \qbezier(120,33)(120,10)(29,0)
%                        % Max and mins
%                        %\thicklines
%                        %\put(20,60){\line(1,0){8}}
%                        \thinlines
%                        %\put(19,61){\makebox(0, 0)[rb]%
%                        %{$\left(\overline{g^p}%
%                        %-\overline{c^p}\right)^*$}}
%                        \put(145,90)%
%                        {\makebox(0,0)[r]
%                        {\shortstack{Optimal Anywhere\\for\\%
%                        $\frac{|c^s|}{\lambda^p} \to
%                        \infty$}}}
%                        \put(96.5,25){\makebox(0,0)[rb]%
%                        {$\eta^* = 0$}}
%                \end{picture}}
%                } % End of \put\scalebox for upper figure
%                %
%                \put(0,0){\scalebox{.5}{
%                \begin{picture}(241,149)(-31,-22)
%                        \thicklines
%                        % Horizontal Axis
%                        \put(-10, 0){\vector(1, 0){155}}
%                        \put(147, 0){\makebox(0, 0)[l]%
%                        {$\var\left(g^p-c^p\right)$}}
%                        % Vertical Axis
%                        \put(24, -15){\vector(0, 1){125}}
%                        \put(24, 111){\makebox(0, 0)[b]%
%                        {$\overline{g^p} - \overline{c^p}$}}
%                        % Reward-to-Variance Line
%                        \thinlines
%                        \put(120, 0){\line(0,1){110}}
%                        \put(126,52.5){\rotatebox{90}{\makebox(0,0)[t]%
%                        {\shortstack%
%                        {Variance maximizer\\for\\$G^T \to \infty$}}}}
%                        \put(22,52.5){\rotatebox{90}{\makebox(0,0)[b]%
%                        {\shortstack%
%                        {Variance minimizer\\for\\$G^T \to -\infty$}}}}
%                        \put(120, 33){\circle*{10}}
%                        \put(24, 0){\circle*{10}}
%                        \put(72,80){\makebox(0,0)%
%                        {$\lim\limits_{G^T\to\pm\infty} \eta^*%
%                        =\mp \infty$}}
%                        % Threshold Value
%                        \thinlines
%                        % Curve, starting from upper left and moving CW
%                        \thinlines
%                        \qbezier(35,35)(35,60)(51,60)
%                        \qbezier(51,60)(80,60)(100,50)
%                        \qbezier(100,50)(120,40)(120,33)
%                        \qbezier(120,33)(120,10)(29,0)
%                        % Max and mins
%                        \thicklines
%                        \put(120,-4){\line(0,1){8}}
%                        \thinlines
%                        \put(120,-5){\makebox(0, 0)[t]%
%                        {$\left(\var\left(g^p-c^p\right)\right)^*$}}
%                \end{picture}}
%        }% End of \put\scalebox for lower figure
%        \end{picture}
%        \label{fig:risk_sensitivity_variance_cycle_limit_case}
%        }
        \caption[Visualization of Reward-to-Variance
        Maximization]{Reward-to-variance maximization. Curves show
        \aimention{Vilfredo Pareto}Pareto frontiers for maximizing net
        gain mean and minimizing net gain variance.  $G^T$ denotes the
        net gain survival threshold.  $\eta^*$ denotes the maximal
        reward-to-variance ratio, and the solutions that achieve this
        ratio are shown as large dots at points of tangency with a line
        of slope $\eta^*$.  The dotted parabolic arrow in
        \subref{fig:risk_sensitivity_variance_cycle_visualization} shows
        how the figure changes as $|c^s|/\lambda^p$ increases. It is not
        necessary that $|\mu^*| < \infty$ and $(\sigma^2)^* < \infty$.}
        \label{fig:risk_sensitivity_variance}
\end{figure}
%
The curve represents the \aimention{Vilfredo Pareto}Pareto frontiers for
maximizing expected net gain and minimizing or maximizing net gain
variance. Notice that the shape of the the curve shown in
\longref{fig:risk_sensitivity_variance_lifetime} is identical to the
shape of the curve in \longref{fig:risk_sensitivity_variance_cycle},
which relates to the analytical tractability gained by using the
reward-to-variance ratio over the reward-to-variability ratio.
Observations made about the reward-to-variability ratio also hold for
the reward-to-variance ratio, and so we omit a discussion of them here.
\index{examples!graphical solutions!reward-to-variance ratio|)indexdef}%
\index{reward-to-variance ratio!graphical optimization|)}% 
\index{reward-to-variance ratio|)}% 

\subsection{Generalized Pareto Tradeoffs}%
\label{sec:alternate_pareto_tradeoffs}\aimention{Vilfredo Pareto}%
\index{Pareto tradeoffs|(}

As discussed, maximization of a rate has the ability to choose solutions
that are \aimention{Vilfredo Pareto}Pareto efficient with respect to
maximization of one objective and maximization or minimization of
another objective, conditioned on the sign of the first objective. The
particular \aimention{Vilfredo Pareto}Pareto efficient solution chosen
depends on the shapes of the optimization functions. In other words, the
relative importance of maximizing one objective and minimizing another
objective is allowed to vary.  Therefore, maximization of a rate is
equivalent to picking a particular relative importance.

In some cases, it may be useful to let the relative importance of
maximization of one objective and minimization of a different objective
be a parameter of the environment. For example, for objectives $A$ and
$B$ and weight $w \in \R$, the maximization of $A - w B$ produces a
\aimention{Vilfredo Pareto}Pareto efficient solution to maximization of
$A$ and minimization of $B$, given that $w$ is the relative importance
of minimizing $B$ over maximizing $A$. If $w < 0$, then $|w|$ represents
the relative importance of \emph{maximizing} $B$ over maximizing $A$.
Maximization of $A - w B$ is sufficient but not necessary for a solution
to be \aimention{Vilfredo Pareto}Pareto efficient with respect to
objectives $A$ and $B$; therefore, unless $A$ and $B$ are convex
functions, there may exist additional \aimention{Vilfredo Pareto}Pareto
efficient solutions that are not accessible by this method.  Either way,
collecting solutions to this maximization problem for all $w \in \R$ is
one way of generating a \aimention{Vilfredo Pareto}Pareto efficient set.

Therefore, here we recast the tradeoffs from
\longref{sec:alternate_rate_tradeoffs} as optimization problems of
relative importance. Keeping in mind that maximization of $A - w B$ is a
sufficient but not necessary condition for \aimention{Vilfredo
Pareto}Pareto efficient solutions, this process can be viewed as picking
a particular point on the curves shown in
\longrefs{fig:rate_maximization}--%
\shortref{fig:risk_sensitivity_variance}.

\subsubsection{Gain and Time}
\index{Pareto tradeoffs!gain and time|(}

Take some weight $w \in \R$ that represents the relative importance of
decreasing total time over increasing net gain (\ie, if $w < 0$, then
$|w|$ is the relative importance of \emph{increasing} total time over
increasing total net gain). Solutions that maximize%
\index{Pareto tradeoffs!gain and time!gain discounted by
time|(indexdef}%
%
\begin{equation}
        \E( G_1 ) - w \E( T_1 )
        \label{eq:payoff_pareto_gain_and_time}
\end{equation}
%
\index{Pareto tradeoffs!gain and time!gain discounted by
time|)indexdef}%
will be \aimention{Vilfredo Pareto}Pareto efficient with respect to
these two objectives. If all of the solutions for all $w \in \R$ are
collected, the result will be a curve like the one shown in
\longref{fig:rate_maximization}. This is analogous to optimizing $\E(
\oft{G}_1 ) - w \E( \oft{T}_1 )$ in a classical \ac{OFT} context.
\Citet{Schoener71} considered net gain maximization and time
minimization as two extremes on a continuum of behaviors that would
maximize reproductive success in a certain environment. If $w \in
\R_{\geq0}$, then this problem captures this idea, where $w$ picks a
particular \aimention{Vilfredo Pareto}Pareto efficient solution for a
certain environment. As discussed by \citet{HouMc99}, when $w$ is set to
the maximum long-term rate of gain (\ie, the maximum value of
$\E(G_1)/\E(T_1)$), the behavior that maximizes
\longref{eq:payoff_pareto_gain_and_time} will also be the behavior that
maximizes the long-term rate of gain.
\index{Pareto tradeoffs!gain and time|)}

\subsubsection{Efficiency}%
\index{Pareto tradeoffs!efficiency|(}

Take some weight $w \in \R$ that represents the relative importance of
decreasing total cost over increasing \emph{gross} gain (\ie, if $w <
0$, then $|w|$ is the relative importance of \emph{increasing} total
cost over increasing total net gain). Solutions that maximize%
\index{Pareto tradeoffs!efficiency!gain discounted by cost|(indexdef}%
%
\begin{equation}
        \E( G_1 + C_1 ) - w \E(C_1)
        \quad \text{or, equivalently,} \quad
        \E(G_1) + (1-w) \E(C_1)
        \label{eq:payoff_pareto_efficiency}
\end{equation}
%
\index{Pareto tradeoffs!efficiency!gain discounted by cost|)indexdef}%
will be \aimention{Vilfredo Pareto}Pareto efficient with respect to
these two objectives. If all of the solutions for all $w \in \R$ are
collected, the result will be a curve like the one shown in
\longref{fig:efficiency_maximization}. 
\index{Pareto tradeoffs!efficiency|)}

\subsubsection{Risk-Sensitivity: Mean and Standard Deviation}%
\index{Pareto tradeoffs!mean and standard deviation|(}%
\index{risk sensitivity|(}

Take some weight $w \in \R$ that represents the relative importance of
decreasing net gain standard deviation\footnote{Again, for simplicity,
we assume \longrefs{eq:cycle_proc_gross_gain_2moment}--%
\shortref{eq:cycle_net_gain_2moment}.} over increasing expected net gain
(\ie, if $w < 0$, then $|w|$ is the relative importance of increasing
net gain standard deviation over increasing expected net gain).
Solutions that maximize%
\index{Pareto tradeoffs!mean and standard deviation!mean discounted by
standard deviation|(indexdef}%
%
\begin{equation}
        \E(G^{N^p}) - w \std(G^{N^p})
        \quad \text{ or, equivalently, } \quad
        \E(G_1) - \frac{w}{\sqrt{N^p}} \std(G_1)
        \label{eq:payoff_pareto_stddev}
\end{equation}
%
\index{Pareto tradeoffs!mean and standard deviation!mean discounted by
standard deviation|)indexdef}%
will be \aimention{Vilfredo Pareto}Pareto efficient with respect to
these two objectives. If all of the solutions for all $w \in \R$ are
collected, the result will be a curve like the one shown in
\longref{fig:risk_sensitivity_sharpe}. 
\index{Pareto tradeoffs!mean and standard deviation|)}

\subsubsection{Risk-Sensitivity: Mean, Variance, and Expected Utility}
\index{Pareto tradeoffs!mean and variance|(}

The analytical tractability of \longref{eq:payoff_pareto_stddev} can be
low. However, substituting variance for standard deviation achieves a
similar goal while allowing for a simpler analysis. That is, take some
weight $w \in \R$ that represents the relative importance of decreasing
net gain variance over increasing expected net gain (\ie, if $w < 0$,
then $|w|$ is the relative importance of increasing net gain variance
over increasing expected net gain). Solutions that maximize%
\index{Pareto tradeoffs!mean and variance!mean discounted by
variance|(indexdef}%
%
\begin{equation}
        \E(G^{N^p}) - w \var(G^{N^p})
        \quad \text{ or, equivalently, } \quad
        \E(G_1) - w \var(G_1)
        \label{eq:payoff_pareto_variance}
\end{equation}
%
\index{Pareto tradeoffs!mean and variance!mean discounted by
variance|)indexdef}%
will be \aimention{Vilfredo Pareto}Pareto efficient with respect to
these two objectives. If all of the solutions for all $w \in \R$ are
collected, the result will be a curve like the one shown in
\longref{fig:risk_sensitivity_variance}.

\index{utility theory|(}This approach was suggested by \citet{Real80} as
a way of applying the mean-variance methods of \citet{Markowitz52} and
\citet{Tobin58} to biology. \Citeauthor{Real80} calls this
\emph{variance discounting}.  The parameter $w$ reflects a shape
parameter of the agent's utility function\footnote{A \emph{utility
function} quantifies the preferences of an agent. For example, a utility
function shaped one way may indicate that an agent is risk prone whereas
a utility function shaped another way may indicate that an agent is risk
avoiding. Economists conventionally assumed that agents make decisions
that maximize future expected utility. The foundations of modern utility
theory are due to \citet{VNM44}; these results are summarized by
\citet{Marschak46}. Utility theory will be discussed further in
\longref{sec:stochastic_dominance}.}.  \Citeauthor{Real80} uses an
approximation of a general utility function to argue that maximizing
\longref{eq:payoff_pareto_variance} leads to maximal expected utility.
\Citet{SK86} make this argument exact by using an approach of
\citet{Caraco80} and some assumptions about the shape of the utility
function and the distribution of returns. In either case, there must be
some justification for the choice of $w$; information needs to be known
about the shape of the agent's utility function. In an engineering
context, this means that $w$ is a parameter of a utility function that
captures the design preferences. The optimization of
\longref{eq:payoff_pareto_variance} should on average lead to the most
preferable outcome with respect to this utility function\footnote{The
ratio-based risk-sensitivity methods in
\longref{sec:alternate_rate_tradeoffs} may be viewed as choosing an
agent's utility function and therefore its preferences based on its
environment. Here, the preferences are a design
variable.}.\index{utility theory|)}%
\index{Pareto tradeoffs!mean and variance|)}
\index{risk sensitivity|)}%
\index{Pareto tradeoffs|)}

\subsection{Constraints}%
\index{constraints|see{optimization constraints}}%
\index{optimization constraints|(}

In \longrefs{sec:alternate_rate_tradeoffs} and
\shortref{sec:alternate_pareto_tradeoffs}, multiple optimization
objectives are traded off in a way to combine elements of both.  In
order to introduce threshold effects, the optimization problems are
ultimately formulated in terms of \emph{excess}. These thresholds could
be made more strict by acting on the means of the distributions rather
than on the outcomes. Thus, for some threshold $\hat{X}$, instead of
maximizing $X - \hat{X}$, we can maximize $X$ subject to the constraint
that $\E(X) \geq \hat{X}$. These type of constraints can be viewed as
drawing horizontal and vertical boundaries on the \aimention{Vilfredo
Pareto}Pareto frontiers in \longrefs{fig:rate_maximization}--%
\shortref{fig:risk_sensitivity_variance}. The optimal solution remains
\aimention{Vilfredo Pareto}Pareto efficient with respect to all
objectives in question; however, it will be optimal with respect to one
particular objective when defined over the constrained domain.
Unfortunately, analytically defining this constrained domain may be
challenging.

\subsubsection{Gain and Time}
\index{optimization constraints!gain and time|(}

Let $T$ be some threshold of time. Consider the objective
%
\begin{equation*}
        \text{maximize } \E(G^{N^p}) 
        \quad \text{ subject to } \quad
        \E(T^{N^p}) \leq T
\end{equation*}
%
This is identical to
%
\begin{equation*}
        \text{maximize } \E(G_1) 
        \quad \text{ subject to } \quad
        \E(T_1) \leq \frac{T}{N^p}
\end{equation*}
%
This is equivalent to drawing a vertical line at $\E(T_1)=T/N^p$ on
\longref{fig:rate_maximization_cycle} and choosing the point on the
\aimention{Vilfredo Pareto}Pareto efficient curve to the left of that
line that has the highest $\E(G_1)$. Notice that as $N^p$ increases, the
feasible set of behaviors decreases, which eventually drives down the
maximum possible net gain.

Similarly, let $G^T$ be some threshold of net gain. Consider the
objective
%
\begin{equation*}
        \text{minimize } \E(T^{N^p}) 
        \quad \text{ subject to } \quad 
        \E(G^{N^p}) \geq G^T
\end{equation*}
%
This is identical to
%
\begin{equation*}
        \text{minimize } \E(T_1) 
        \quad \text{ subject to } \quad
        \E(G_1) \geq \frac{G^T}{N^p}
\end{equation*}
%
This is equivalent to drawing a horizontal line at $\E(G_1)=G^T/N^p$ on
\longref{fig:rate_maximization_cycle} and choosing the point on the
\aimention{Vilfredo Pareto}Pareto efficient curve above that line that
has the lowest $\E(T_1)$.  Notice that as $N^p$ increases, the feasible
set of behaviors increases, which eventually deactivates the
constraint\footnote{Borrowing language from \aimention{Joseph-Louis
Lagrange}Lagrange multiplier methods \citep{Bertsekas95}, a constraint
is not \emph{active} when it is satisfied with strict inequality.
Constraints form boundaries around solutions; when a solution falls on a
boundary, the constraint is said to be active, which implies that the
optimal solution would most likely fall outside of the boundary if it
were removed.}.
\index{optimization constraints!gain and time|)}

\subsubsection{Efficiency}
\index{optimization constraints!gain and cost|(}

Let $C^T$ be some cost threshold. Consider the objective
%
\begin{equation*}
        \text{maximize } \E(G^{N^p})+\E(C^{N^p}) 
        \quad \text{ subject to } \quad
        \E(C^{N^p}) \leq C^T
\end{equation*}
%
This is identical to
%
\begin{equation*}
        \text{maximize } \overline{g^p}
        \quad \text{ subject to } \quad
        \E(C_1) \leq \frac{C^T}{N^p}
\end{equation*}
%
This is equivalent to drawing a vertical line at $\E(C_1)=C^T/N^p$ on
\longref{fig:efficiency_maximization_cycle} and choosing the point on
the \aimention{Vilfredo Pareto}Pareto efficient curve to the left of
that line that has the highest $\overline{g^p}$. Notice that as $N^p$
increases, the feasible set of behaviors decreases, which eventually
drives down the maximum possible gross gain.

Similarly, let $G_g^T$ be some threshold of \emph{gross} gain. Consider
the objective
%
\begin{equation*}
        \text{minimize } \E(C^{N^p}) 
        \quad \text{ subject to } \quad
        \E(G^{N^p})+\E(C^{N^p}) \geq G_g^T
\end{equation*}
%
This is identical to
%
\begin{equation*}
        \text{minimize } \E(C_1) 
        \quad \text{ subject to } \quad
        \overline{g^p} \geq \frac{G_g^T}{N^p}
\end{equation*}
%
This is equivalent to drawing a horizontal line at
$\overline{g^p}=G_g^T/N^p$ on
\longref{fig:efficiency_maximization_cycle} and choosing the point on
the \aimention{Vilfredo Pareto}Pareto efficient curve above that line
that has the lowest $\E(C_1)$. Notice that as $N^p$ increases, the
feasible set of behaviors increases, which eventually deactivates the
constraint.
\index{optimization constraints!gain and cost|)}

\subsubsection{Risk-Sensitivity: Mean, Standard Deviation, and
Variance}%
\index{optimization constraints!mean and variance|(}

Let $\hat{\sigma}^2$ be some threshold of net gain variance. Consider
the objective
%
\begin{equation*}
        \text{maximize } \E(G^{N^p}) 
        \quad \text{ subject to } \quad
        \var(G^{N^p}) \leq \hat{\sigma}^2
\end{equation*}
%
This is identical to
%
\begin{equation*}
        \text{maximize } \E(G_1) 
        \quad \text{ subject to } \quad
        \var(G_1) \leq \frac{\hat{\sigma}^2}{N^p}
\end{equation*}
%
This is equivalent to drawing a vertical line at
$\var(T_1)=\hat{\sigma}^2/N^p$ on
\longref{fig:risk_sensitivity_variance_cycle} and choosing the point on
the \aimention{Vilfredo Pareto}Pareto efficient curve to the left of
that line that has the highest $\E(G_1)$. Notice that as $N^p$
increases, the feasible set of behaviors decreases, which eventually
drives down the maximum possible net gain.

Similarly, let $G^T$ be some threshold of net gain. Consider the
objective
%
\begin{equation*}
        \text{minimize } \var(G^{N^p}) 
        \quad \text{ subject to } \quad
        \E(G^{N^p}) \geq G^T
\end{equation*}
%
This is identical to
%
\begin{equation*}
        \text{minimize } \var(G_1) 
        \quad \text{ subject to } \quad
        \E(G_1) \geq \frac{G^T}{N^p}
\end{equation*}
%
This is equivalent to drawing a horizontal line at $\E(G_1)=G^T/N^p$ on
\longref{fig:risk_sensitivity_variance_cycle} and choosing the point on
the \aimention{Vilfredo Pareto}Pareto efficient curve above that line
that has the lowest $\var(G_1)$. Notice that as $N^p$ increases, the
feasible set of behaviors increases, which eventually deactivates the
constraint.%
\index{optimization constraints!mean and variance|)}

\index{optimization constraints!mean and standard deviation|(}%
\paragraph{Standard Deviation:} The optimization objectives here could
be rewritten using standard deviation. That is, variance could be
replaced with standard deviation and any constraints on variance could
be replaced with its square root. The resulting optimization objective
leads to the same solutions as the original objective. Therefore, we
omit a special standard deviation case. Often, the use of variance will
improve analytical tractability of these problems.
\index{optimization constraints!mean and standard deviation|)}%
\index{solitary agent model!processing-only analysis!optimization|)}
\index{optimization constraints|)}

\section{Future Directions Inspired by PMPT}%
\label{sec:future_econ_directions}%
\index{future directions}%
\index{finance!post-modern portfolio theory (PMPT)|(}

\index{MVA|see{mean-variance analysis}}As with classical \ac{OFT},
results here are influenced primarily by \ac{MPT}, which describes a set
of methods for portfolio investment and capital budgeting that follows
from the work of \citet{Markowitz52,Markowitz59} and \citet{Tobin58} and
applications of that work by \citet{Lintner65}, \citet{Mossin66}, and
\citet{Sharpe64}.  \ac{MPT} is mostly concerned with
\acro[\index{mean-variance analysis (MVA)}mean-variance analysis
(MVA)]{MVA}{\index{mean-variance analysis (MVA)"|indexglo}mean-variance
analysis}.  Proponents of \ac{MPT} openly acknowledge that \ac{MVA} is
often too naive \citep{Markowitz59,Sharpe64}; however, its use is
justified because it is well-understood and has low computational
demands.  However, modern technological and theoretical advances may
render these justifications invalid \citep{RF94}. Thus, \ac{PMPT} seeks
to improve upon \ac{MPT} by using more advanced analytical approaches.
We discuss two \ac{PMPT} topics here that may have potential
applications in biology and engineering in the analysis and design of
agents.

\subsection{Lower Partial Moments}

\index{stochasticity!statistics!semivariance|see{stochasticity,
statistics, lower-partial
variance}}\index{semivariance|see{stochasticity, statistics,
lower-partial variance}}\index{LPV|see{stochasticity, statistics,
lower-partial variance}}\index{lower-partial variance|see{stochasticity,
statistics, lower-partial variance}}\index{LPM|see{stochasticity,
statistics, lower-partial moment}}\index{lower-partial
moment|see{stochasticity, statistics, lower-partial moment}}We have used
the well-known constructs of standard deviation and variance to quantify
the variability of a return. However, for a given gain threshold,
minimization of the variation \emph{below} that threshold is much more
important than the variation above the threshold.  That is, rather than
seeking certainty in future gains, it may be more useful to minimize the
uncertainty in negative gain budgets. Therefore, \citet{Markowitz59}
introduces the \emph{semivariance} which \citet{Bawa75} calls the
\emph{\acro[\index{stochasticity!statistics!lower-partial variance
(LPV)|(indexdef}lower-partial
variance~(LPV)]{LPV}{\index{stochasticity"!statistics"!lower-partial
variance (LPV)"|indexglo}lower partial variance}}. The \ac{LPV} is the
expected value of the squared negative deviations of possible outcomes
from some point of reference. That is, for a distribution $F$ and a
reference point $t$, the \ac{LPV} is defined by
%
\begin{equation*}
        \LPV(t;F)
        \triangleq \int_{x}^t (t-x)^2 \total F(x)
\end{equation*}
%
where the \index{mathematics!functions!integral ($\int$)}integral is the
\aimention{Henri L. Lebesgue}Lebesgue
integral.\index{stochasticity!statistics!lower-partial variance
(LPV)|)indexdef} \Citet{BL77} generalize this idea with the notion of an
$n$\th{}\ order
\emph{\acro[\index{stochasticity!statistics!lower-partial moment
(LPM)|(indexdef}lower-partial
moment~(LPM)]{LPM}{\index{stochasticity"!statistics"!lower-partial
moment (LPM)"|indexglo}lower-partial moment}} defined by
%
\begin{equation*}
        \LPM_n(t;F)
        \triangleq \int_{x}^t (t-x)^n \total F(x)
\end{equation*}
%
\index{stochasticity!statistics!lower-partial moment
(LPM)|)indexdef}Clearly, $\LPV(t;F) = \LPM_2(t;F)$ for all $t \in \R$.
These \emph{lower-partial} moments\footnote{\Citet{Bawa78} shows that
for all $n \in \N$ and $t \in \R$, $\LPM_n(t;F)$ is a convex function of
$F$.} are generalized asymmetric notions of the familiar central moments
(\eg, variance and skewness).  Rather than integrating (\ie, summing)
all variations around the mean of a distribution, the \ac{LPM}
integrates all variations that fall beneath some arbitrary reference
which might be viewed as a target benchmark (\eg, a net gain threshold).
This is a type of \emph{downside risk} measure \citep{GH99,RF94}.

\index{MLPM|see{mean-lower-partial-moment
analysis}}\index{MSA|see{mean-lower-partial-moment
analysis}}\index{mean-semivariance
analysis|see{mean-lower-partial-moment
analysis}}\index{mean-lower-partial-moment (MLPM) analysis|(indexdef}The
$n$\th{}\ order \ac{LPM} can be substituted for variance in \ac{MVA} to
yield a new method of analysis that trades off greater expected returns
and shortfall uncertainty without putting any pressure on variances
above return benchmarks. This is called
\emph{\acro{MLPM}{mean-lower-partial-moment}} analysis
\citep{Bawa78,LR88}. \index{mean-lower-partial-variance
analysis|(indexdef}\index{mean-semivariance
analysis|see{mean-lower-partial-variance analysis}}When $n=2$, this is
called \emph{\acro{MLPV}{mean-lower-partial-variance}} analysis
\citep{BL77} or \emph{\acro{MSA}{mean-semivariance analysis}}
\citep{Mao70}. When \ac{MLPV} analysis is applied to normal
distributions, the results are identical to \ac{MVA}\footnote{This is
due to the symmetry of normal
distributions.}.\index{mean-lower-partial-variance analysis|)indexdef}
In fact, \ac{MLPM} analysis will always have results that ``do at least
as well'' as \ac{MVA} \citep{BL77}. Thus, \citet{Mao70} claims that
``the time has come to shift'' from \ac{MVA} to \ac{MSA} in capital
budgeting. We believe that optimization based on these new downside risk
frameworks may also lead to advances in behavioral analysis and solitary
agent design.\index{mean-lower-partial-moment (MLPM) analysis|)indexdef}

\subsection{Stochastic Dominance}
\label{sec:stochastic_dominance}%
\index{finance!stochastic dominance (SD)|(indexdef}

\index{SD|see{finance, stochastic dominance}}\index{stochastic
dominance|see{finance, stochastic dominance}}\index{stochasticity!random
variable|(}\index{stochasticity!distributions|(}\index{investment
returns|see{finance, investment returns}}\index{finance!investment
returns}Investment returns (\eg, net gain) are naturally random
variables. Thus, \index{portfolios|see{finance,
portfolios}}\index{finance!portfolios}portfolios or \index{capital
budgeting|see{finance, capital budgets}}\index{finance!capital
budgets}budgets (\eg, processing probability and time choices)
correspond to random variables with different probability distributions.
Portfolio selection is thus a decision problem of choosing the most
preferable probability distribution. \index{maximal utility|see{utility
theory}}\index{maximal expected utility|see{utility
theory}}\index{utilitarianism|see{utility theory}}\index{expected
utility|see{utility theory}}\index{von Neumann-Morgenstern
utility|see{utility theory}}\index{utility theory|(}\Citet{VNM44}
characterize preferences in terms of utility functions, which represent
the effective value of certain returns to an agent. Different-shaped
utility functions correspond to different preferences, and rational
agents\footnote{\index{rational agent|see{utility theory}}\Citet{VNM44}
define what \emph{rational agent} means with a number of axioms about
preference.} will maximize their expected utility. That is, if $F$ and
$G$ are two distributions of returns and $u$ is an agent's utility
function, the distribution $F$ is preferred over the distribution $G$ if
%
\begin{equation}
        \int_{-\infty}^\infty u(x) \total F(x)
        - \int_{-\infty}^\infty u(x) \total G(x)
        > 0
        \label{eq:maximal_exp_utility}
\end{equation}
%
This rule can be used to pick the best distribution(s) from a set of
distributions that correspond to portfolio
choices.\index{stochasticity!distributions|)}\index{stochasticity!random
variable|)} While each of these probability distributions may be known
or at least knowable, it may not be possible to completely describe an
agent's \index{utility function|see{utility theory}}utility function.
Therefore, it is desirable to define rules like
\longref{eq:maximal_exp_utility} that are a function of \emph{only} the
probability distributions. Distributions that are preferable with
respect to these rules will return maximal utility for \emph{all}
utility functions that share certain very general characteristics. These
are known as \emph{\acro[stochastic
dominance~(SD)]{SD}{\index{finance"!stochastic dominance
(SD)"|indexglo}stochastic dominance}} rules.  \index{finance!stochastic
dominance (SD)!first-order (FSD)|(indexdef}For example, as shown by
\citet{Bawa75}, for any two distributions $F$ and $G$, $F$ is preferred
(\ie, results in a greater expected utility) to $G$ for any increasing
and continuously differentiable utility function $u(x)$ assumed to have
finite values for finite values of $x$ \emph{if and only if} there
exists some $x_0 \in \R$ such that
%
\begin{equation}
        F(x) \leq G(x) \text{ for all } x \in \R
        \quad \text{ and } \quad
        F(x_0) < G(x_0)
        \label{eq:fsd}
\end{equation}
%
\longref{eq:fsd} is called \emph{\acro[first-order stochastic
dominance~(FSD)]{FSD}{\index{finance"!stochastic dominance
(SD)"!first-order (FSD)"|indexglo}first-order stochastic dominance}}.
\ac{FSD} implies that for any return benchmark, the probability of
failing to meet that benchmark will be less with the $F$
distribution\footnote{Of course, \ac{FSD} provides no guarantee that a
particular outcome from the $G$ distribution will not succeed when the
outcome from the $F$ distribution will fail. Dominance for all outcomes
is the strongest form of \ac{SD} (and is rarely used).}. This class of
utility functions for \ac{FSD} is very broad.\index{finance!stochastic
dominance (SD)!first-order (FSD)|)indexdef} There are higher-order
\ac{SD} rules that apply to smaller sets of utility functions (\eg, risk
averse utility functions).  In particular, \citet{Bawa75} shows that
\emph{\acro[\index{finance!stochastic dominance (SD)!third-order
(TSD)}third-order stochastic
dominance~(TSD)]{TSD}{\index{finance"!stochastic dominance
(SD)"!third-order (TSD)"|indexglo}third-order stochastic dominance}} has
an important relationship to \ac{MLPM} analysis. An extensive
categorized bibliography of articles on \ac{SD} and its application is
given by \citet{Bawa82}, who also briefly discusses the foundations of
\ac{SD}. 

Biologists have recognized the validity of utility functions
\citep[\eg,][]{Caraco80,Real80,SK86}. However, the methods of \ac{SD}
have not been applied to compare alternative behaviors in terms of
maximal expected utility in a convincing way. That is, \ac{MVA} is used
to approximate an \ac{SD} analysis; however, there seems to be little
recognition that this approximation is only valid for a limited set of
utility functions. \Citet{Bawa75} shows that for a large set of utility
functions, \ac{MVA} meets neither necessary nor sufficient conditions
for optimality and \ac{MLPV} analysis should be used instead.  As design
preferences are often rational, engineering can also benefit from an
\ac{SD} approach. If nothing else, \ac{SD} provides a deeper
understanding of the consequences of behavioral choice by avoiding
heuristic arguments and approximations. Additionally, using \ac{SD}
promises to allow both engineers and behavioral ecologists to benefit
not only from classical works in economics but also from the modern-day
research in the field.\index{utility theory|)}%
\index{finance!stochastic dominance (SD)|)indexdef}%
\index{finance!post-modern portfolio theory (PMPT)|)}
