A new multi-objective optimization algorithm to handle problems that are hightly constrained, highly nonlinear, and with mixed types of design variables

1. A New Multi-Objective Mixed-Discrete
Particle Swarm Optimization Algorithm
(MO-MDPSO)
Weiyang Tong*, Souma Chowdhury#, and Achille Messac#
* Syracuse University, Department of Mechanical and Aerospace Engineering
# Mississippi State University, Department of Aerospace Engineering
ASME 2014 International Design Engineering Technical Conference
August 18-20, 2014
Buffalo, NY
For citations, please refer to the journal version of this paper,
by Tong et al., "A Multi-Objective Mixed-Discrete Particle Swarm Optimization with Multi-Domain
Diversity Preservation", Structural and Multidisciplinary Optimization. DOI:10.1007/s00158-015-1319-8

2. Particle Swarm Optimization
2
• Particle Swarm Optimization (PSO)
• was introduced by Eberhart and Kennedy in 1995
• was inspired by the swarm behavior observed in nature
• is a population based stochastic algorithm
• Advantages of basic PSO:
• Fast convergence
• Easy implementation
• Few parameters to adjust
PSO suffers from pre-stagnation, which is mainly attributed to
the loss of diversity during the fast convergence
Powerful optimizer for single objective
unconstrained continuous problems

3. Single Objective Mixed-Discrete PSO*
Position update:
𝒙𝑖 𝑡 + 1 = 𝒙𝑖 𝑡 + 𝒗𝑖 𝑡 + 1
Velocity update:
𝒗𝑖 𝑡 + 1 = 𝑤𝒗𝑖 𝑡 + 𝑟1 𝐶1 𝑃𝑖
𝑙
(𝑡) − 𝒙𝑖 + 𝑟2 𝐶2 𝑃 𝑔(𝑡) − 𝒙𝑖 + 𝑟3 𝛾𝑐 𝒗𝑖(𝑡)
𝑤 – inertia weight
𝒙𝑖 – position of a particle
𝒗𝑖 – velocity of a particle
𝐶1 – cognitive parameter
𝐶2 – social parameter
𝑡 – generation
𝑟 – random number between 0-1
𝑃𝑖
𝑙
– pbest of a particle
𝑃 𝑔
– gbest of the swarm
3
Diverging velocity
Diversity
preservation
coefficient
Inertia Local search Global search
*: Chowdhury et al (2013)

4. Outline
• Research Motivation and Objectives
• Search Strategy in MO-MDPSO
• Dynamics of MO-MDPSO
• Multi-domain Diversity Preservation
• Numerical Experiments
• Continuous Unconstrained Test Problems
• Continuous Constrained Test Problems
• Mixed-Discrete Test Problems
• Concluding Remarks
4

5. Research Motivation
5
• Major attributes involved in complex engineering optimization problems:
• Nonlinearity
• Multimodality
• Constraints
• Mixed-discrete
• Multi-objective
Constrained Multiobjective
MultimodalNonlinear
Mixed types of
variables
MDPSO
addressed by

6. Research Objective
• Develop a Multi-Objective MDPSO (MO-MDPSO)
• Introduce the Multi-objective capability to MDPSO
• Allow the algorithm to search for Pareto solutions
• Make important advancements on the diversity preservation
technique – primary feature of MDPSO
• Avoid stagnation and capture the complete Pareto frontier
• Keep a desirably even distribution of Pareto solutions
6

7. Search Strategies in Multi-Objective PSO
7
• MOPSO by Parsoupulos and Vrahatis (2002)
• DNPSO by Hu and Eberhart (2002)
• NSPSO by Li (2003)
• MOPSO by Coello (2004)
• To best retain the original dynamics, the Pareto based strategy is selected
Aggregating function based
Single objective based
Hybrid with other techniques
Pareto dominance based
Basic PSO MOPSO
Local
leader
The best solution is based on a
particle’s own history
Local set with non-dominated
historical solutions are found
within the local neighborhood
Global
leader
The best solution among all
local best solutions
The global Pareto solutions are
obtained using all local ones

8. Solutions Comparison
if both solu-x and solu-y are infeasible
choose the one with the smaller constraint violation;
else if solu-x is feasible and solu-y is infeasible
choose solu-x;
else if solu-x is infeasible and solu-y is feasible
choose solu-y;
else both solu-x and solu-y are feasible
apply non-dominance comparison:
 solu-x strongly dominates solu-y if and only if
𝑓𝑘 𝑥 < 𝑓𝑘 𝑦 , ∀k = 1,2,…, N
 solu-x weakly dominates solu-y if
𝑓𝑘 𝑥 ≤ 𝑓𝑘 𝑦 for at least one k
 solu-x is non-dominated with solu-y when
𝑓𝑜𝑟 𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝑘: 𝑓𝑘 𝑥 > 𝑓𝑘 𝑦 , for the rest: 𝑓𝑘 𝑥 < 𝑓𝑘 𝑦
8

9. 1
2
3
4
5
1
2
3
4 5
f1
f2
0 0.5 1 1.5 2 2.5 3
0
0.5
1
1.5
2
2.5
3
Infeasible
R
egion
Actual Boundary
Local
Pareto set
Dynamics of MO-MDPSO
Position update:
𝒙𝑖 𝑡 + 1 = 𝒙𝑖 𝑡 + 𝒗𝑖 𝑡 + 1
Velocity update:
𝒗𝑖 𝑡 + 1 = 𝑤𝒗𝑖 𝑡 + 𝑟1 𝐶1 𝑷𝑖
𝑙
(𝑡) − 𝒙𝑖 + 𝑟2 𝐶2 𝑷𝑖
𝑔
(𝑡) − 𝒙𝑖 + 𝑟3 𝛾 𝑐,𝑖 𝒗𝑖(𝑡)
9
𝑷𝑖
𝑙
– local leader of particle-i, which is
the selected from the local Pareto set
𝑷𝑖
𝑔
– global leader of particle-i that is
determined by a stochastic process
Crowding Distance – to manage the
size of local/global Pareto set
Multiple global leaders
Inertia Local search Global search
Applied w.r.t.
each particle’s
global leader
Current particle Stored particle

10. Diversity Metrics
The diversity is measured based on the spread in design space
for continuous variables:
𝐷𝑐 =
𝑗=𝑚+1
𝑛
𝑥 𝑚𝑎𝑥,𝑗(𝑡) − 𝑥 𝑚𝑖𝑛,𝑗(𝑡)
𝑋 𝑚𝑎𝑥,𝑗 − 𝑋 𝑚𝑖𝑛,𝑗
1
𝑛−𝑚
for discrete variables:
𝐷 𝑑
𝑗
=
𝑥 𝑚𝑎𝑥,𝑗(𝑡) − 𝑥 𝑚𝑖𝑛,𝑗(𝑡)
𝑋 𝑚𝑎𝑥,𝑗 − 𝑋 𝑚𝑖𝑛,𝑗
10
Considering the impact of outlier solutions, the diversity metrics are modified as
𝐷𝑐,𝑖 = 𝐹𝜆𝑖 𝐷𝑐, and 𝐷 𝑑,𝑖
𝑗
= 𝐹𝜆𝑖 𝐷 𝑑
𝑗
where
𝐹𝜆𝑖 = 𝜆
𝑁 𝑝 + 1
𝑁𝑖 + 1
1
𝑛
𝜆 is used to define the fractional domain w.r.t. the global leader of particle-i
The spread along
the jth dimension
The upper & lower bounds
along the jth dimension
Number of
candidate solutions
per global leader
Number of candidate
solutions enclosed by
the 𝜆-fractional domain

11. Hypercube enclosing
72 candidate solutions
X1
X2
0 1 2 3
0
1
2
3
Particles
Global leaders
Multiple λ-Fractional Domains
11
• 7 global leaders are
observed
• Ideally, each fractional
domain should enclose
10 particles
• Particles located in the
overlapping regions are
uniformly re-allocated
between domains
13
10
14
𝜆 = 0.25
Design Variable Space

12. Multi-domain Diversity Preservation
The boundaries of the 𝜆𝑖 - fractional domain are defined as
𝑥𝑖
𝑚𝑎𝑥,𝑗
= 𝑚𝑎𝑥
𝑥 𝑚𝑖𝑛,𝑗 + 𝜆𝑖 𝛿𝑥 𝑗,
𝑚𝑖𝑛 𝑷𝑖
𝑔,𝑗
+
1
2
𝜆𝑖 𝛿𝑥 𝑗, 𝑥 𝑚𝑎𝑥,𝑗
𝑥𝑖
𝑚𝑖𝑛,𝑗
= 𝑚𝑖𝑛
𝑥 𝑚𝑎𝑥,𝑗 − 𝜆𝑖 𝛿𝑥 𝑗,
𝑚𝑎𝑥 𝑷𝑖
𝑔,𝑗
−
1
2
𝜆𝑖 𝛿𝑥 𝑗, 𝑥 𝑚𝑖𝑛,𝑗
12
• Continuous variables
𝛾 𝑐,𝑖 = 𝛾 𝑐0 𝑒𝑥𝑝 −
1
2
𝐷𝑐,𝑖
𝜎𝑐
2
,
𝜎𝑐 =
1
2 ln 1 𝛾 𝑚𝑖𝑛
• Discrete variables
𝛾𝑑,𝑖
𝑗
= 𝛾 𝑑0 𝑒𝑥𝑝 −
1
2
𝐷 𝑑,𝑖
𝑗
𝜎 𝑑
2
,
𝜎 𝑑 =
1
2 ln 1 𝑀 𝑗

13. • Three classes of test problems are used to evaluate the performance of
MO-MDPSO (two performance metrics and comparison with NSGA-II)
• Each of the test problems is run 30 times to compensate for the impact of
random parameters
• Sobol’s quasirandom sequence generator is used for the initial population
User-defined parameters in MO-MDPSO
Numerical Experiment
13
Parameter
Continuous uncon-
strained problems
Continuous con-
strained problems
Mixed-discrete
constrained problems
𝑤 0.5 0.5 0.5
𝐶1 1.5 1.5 1.5
𝐶𝑐0 1.5 1.5 1.5
𝛾 𝑐0 1.0 1.0 1.0
𝛾 𝑚𝑖𝑛 1e-4 1e-6 1e-8
𝛾 𝑑0 NA NA 0.5,1.0
𝜆 0.2 0.1 0.1
Local set size 5 6 10
Global set size 50 50 Up to 100
Population size 100 100 min(5n, 100)

14. Performance Metric for Continuous Unconstrained Problems
Results: Continuous Unconstrained Problems
14
Function
name
Accuracy 𝜸 Diversity ∆
Mean 𝜇 𝛾 Std. deviation 𝜎𝛾 Mean 𝜇∆ Std. deviation 𝜎∆
MO-MDPSO NSGA-II MO-MDPSO NSGA-II MO-MDPSO NSGA-II MO-MDPSO NSGA-II
Fonseca 2 0.0043 0.0004 0.7900 0.0062
Coello 0.0069 0.0009 0.7101 0.0137
Schaffer 1 0.0086 0.0010 0.7131 0.0066
Schaffer 2 0.0116 0.0008 0.9655 0.0015
ZDT 1 0.0009 0.0009 0.0001 0 0.7265 0.4633 0.0045 0.0416
ZDT 2 0.0007 0.0008 7.3e-5 0 0.7248 0.4351 0.0049 0.0246
ZDT 3 0.0042 0.0434 0.0004 4.2e-5 0.7457 0.5756 0.0052 0.0051
ZDT 4 1.5922 0.5130 2.1411 0.1184 0.8306 0.7026 0.1040 0.06465
ZDT 6 0.0337 0.2965 0.0525 0.0131 0.8860 0.6680 0.2072 0.0099

15. Comparison of Performance Indicators
15
Performance of Accuracy
Performance of Diversity
ZDT 4
0
1
2
3
ZDT 3
0
0.01
0.02
0.03
0.04
0.05
ZDT 1
0
0.0002
0.0004
0.0006
0.0008
0.001
0.0012
0.0014
ZDT 2
0
0.0002
0.0004
0.0006
0.0008
0.001
ZDT 6
0
0.1
0.2
0.3
0.4MO-
MDPSO
NSGA2
MO-
MDPSO NSGA2
MO-
MDPSO
NSGA2
MO-
MDPSO
NSGA2
MO-
MDPSO
NSGA2
ZDT 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
ZDT 2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
ZDT 3
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
ZDT 4
0
5
10
ZDT 6
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
MO-
MDPSO
NSGA2
MO-
MDPSO
NSGA2
MO-
MDPSO
NSGA2
MO-
MDPSO
NSGA2
MO-
MDPSO
NSGA2
The lower the better

16. Plots: Continuous Unconstrained Problems
16
Number of function
evaluations for these
problems is 10,000
(25,000 was used by
NSGA-II)
f1
f2
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0 0.2 0.4 0.6 0.8 1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
ZDT 1
30 design variables
ZDT 2
30 design variables
ZDT 3
30 design variables
f1
f2
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
ZDT 4
10 design variables
ZDT 6
10 design variables

17. Plots: Continuous Constrained Problems
17f1
f2
0 50 100 150 200
-200
-150
-100
-50
0 Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0 0.2 0.4 0.6 0.8 1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0.4 0.6 0.8 1
0
1
2
3
4
5
6
7
8
9
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0 20 40 60 80 100 120
10
20
30
40
50
Pareto solution by MO-MDPSO
Actual Pareto solution
BNH
2 design variables
CONSTR
2 design variables
KITA
2 design variables
SRN
2 design variables
TNK
2 design variables
Number of function
evaluations for these
problems is 10,000
(20,000 was used by
NSGA-II)

18. • The MINLP problem
adopted from Dimkou
and Papalexandri*
18
No. of design variables 6
No. of discrete variables 3 (binary)
Function evaluations 10,000
Population size 100
Elite size of NSGA-II 78
Elite size of MO-MDPSO 100
*: Dimkou and Papalexandri (1998)
Mixed-Discrete Constrained Test Problem 1
f1
f2
-60 -50 -40 -30 -20 -10 0
-20
0
20
40
60
80
100
Pareto solution by NSGA-II
Pareto solution by MO-MDPSO

19. 19
• The design of disc brake
problem adopted from
Osyczka and Kundu*
No. of design variables 4
No. of discrete variables 1 (integer)
Function evaluations 10,000
Population size 100
Elite size of NSGA-II 87
Elite size of MO-MDPSO 100
*: Osyczka and Kundu (1998)
Mixed-Discrete Constrained Test Problem 2

20. Multi-objective Wind Farm Optimization
20
• Two objectives:
• Maximize the wind farm
capacity factor
• Minimize the unit land usage
• 150 design variables:
• 100 continuous variables:
Location of turbines
• 50 discrete variables: Type
of turbines
• 1225 constraints:
• Inter-turbine spacing
50 turbines with 10
turbine selections

21. Concluding Remarks
• We developed a new MO-MDPSO algorithm that is capable of handling
all the major attributes in complex engineering optimization problems
• The original dynamics of basic PSO is best retained by introducing the
Pareto dominance strategy to MDPSO
• The multi-domain diversity preservation technique was developed to
maintain a desirably even distribution of Pareto solutions
• MO-MDPSO showed favorable results in solving continuous bi-objective
optimization problems; the performance in solving mixed-discrete
problems is comparable with or better than NSGA-II
• Future work should test MO-MDPSO to high dimensional engineering
problems, and include problems with more than two objectives.
21

22. Acknowledgement
• I would like to acknowledge my research adviser
Prof. Achille Messac, and my co-adviser Dr.
Souma Chowdhury for their immense help and
support in this research.
• Support from the NSF Awards is also
acknowledged.
22

23. 23
Thank you
Test function

24. Coello Fonseca 2
f1
f2
0 0.2 0.4 0.6 0.8 1
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Pareto solution by MO-MDPSO
Actual Pareto solution
Continuous Unconstrained Test Problems
24
Schaffer 1 Schaffer 2
f1
f2
0 1 2 3 4
0
1
2
3
4
Pareto solution by MO-MDPSO
Actual Pareto solution
f1
f2
-1 -0.5 0 0.5 1
0
5
10
15
20
Pareto solution by MO-MDPSO
Actual Pareto solution
Number of function
evaluations for these
problems is 2,000

25. BNH
0
0.05
0.1
0.15
0.2
CONSTR
0
0.002
0.004
0.006
0.008
0.01
KITA
0
0.002
0.004
0.006
0.008
0.01
0.012
SRN
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
TNK
0
0.002
0.004
0.006
0.008
0.01
0.012
Results: Continuous Constrained Problems
25
Function
name
Accuracy 𝜸 Diversity ∆
Mean 𝜇 𝛾 Std. deviation 𝜎𝛾 Mean 𝜇∆ Std. deviation 𝜎∆
BNH 0.1342 0.0204 0.7778 0.0088
CONSTR 0.0070 0.0007 0.8336 0.0101
KITA 0.0091 0.0008 0.7656 0.0086
SRN 0.4909 0.0567 0.7165 0.0046
TNK 0.0055 0.0006 0.9010 0.0216
Performance Metric for Continuous Constrained Problems
CONSTR
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
BNH
0
0.1
0.2
0.3
0.4
0.5
KITA
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
SRN
0
0.2
0.4
0.6
0.8
1
TNK
0
0.2
0.4
0.6
0.8
1
1.2
Performance of Accuracy
Performance of Diversity

26. Illustration of MO-MDPSO
1. Both solu-x and solu-y are
infeasible – choose the one with the
smaller constraint violation;
2. solu-x is feasible and solu-y is
infeasible – choose solu-x;
3. solu-x is infeasible and solu-y is
feasible – choose solu-y;
4. Both solu-x and solu-y are
feasible – apply non-dominance
comparison:
 solu-x strongly dominates solu-y
if and only if:
𝑓𝑘 𝑥 < 𝑓𝑘 𝑦 , ∀k = 1,2,…, N
 solu-x weakly dominates solu-y if:
𝑓𝑘 𝑥 ≤ 𝑓𝑘 𝑦 for at least one k
26
1
2
3
4
5
f1
f2
0 0.5 1 1.5 2 2.5 3
0
0.5
1
1.5
2
2.5
3
Infeasible
R
egion
Actual Boundary
1
2
3
4
5
1
2
3
4 5
f1
f2
0 0.5 1 1.5 2 2.5 3
0
0.5
1
1.5
2
2.5
3
Infeasible
R
egion
Actual Boundary
Current particle Stored particle
Local
Pareto set
A solution is non-dominated when
𝑓𝑜𝑟 𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝑘: 𝑓𝑘 𝑥 > 𝑓𝑘 𝑦 ,
for the rest: 𝑓𝑘 𝑥 < 𝑓𝑘 𝑦
k = 1,2,…, N
The crowding distance is used
to manage the size of Pareto set

27. Multi-directional Diversity Preservation (cont’d)
• Continuous variables
𝛾 𝑐,𝑖 = 𝛾 𝑐0 𝑒𝑥𝑝 −
1
2
𝐷𝑐,𝑖
𝜎𝑐
2
,
𝜎𝑐 =
1
2 ln 1 𝛾 𝑚𝑖𝑛
• The diversity coefficient for
continuous variables is to apply a
repulsion away from each of the
global leaders
• Discrete variables
𝛾𝑑,𝑖
𝑗
= 𝛾 𝑑0 𝑒𝑥𝑝 −
1
2
𝐷 𝑑,𝑖
𝑗
𝜎 𝑑
2
,
𝜎 𝑑 =
1
2 ln 1 𝑀 𝑗
• A stochastic update process is
applied to help particles jump out of
the local hypercude:
 if 𝑟 > 𝛾𝑑,𝑖
𝑗
, use the nearest
vertex approach (NVA)
 else, update randomly to the
upper or lower bound of the
local hypercube
27
The diversity coefficient is expressed as a monotonically
decreasing function of the current diversity metric
Size of feasible
values of the jth
variable

28. Discrete Variables and Constraints Handling
• The Nearest Vertex Approach (NVA)
is used to deal with discrete variables,
where a local hypercube is defined as
𝑯 = 𝑥 𝐿1, 𝑥 𝑈1 , 𝑥 𝐿2, 𝑥 𝑈2 , … , 𝑥 𝐿 𝑚, 𝑥 𝑈 𝑚
and 𝑥 𝐿 𝑗 < 𝑥 𝑗 < 𝑥 𝑈 𝑗, ∀𝑗 = 1,2, … , 𝑚
• The net constraint is used to handle
constraints, as given by
𝑓𝑐 =
𝑝=1
𝑃
max 𝑔 𝑞, 0 +
𝑞=1
𝑄
max ℎ 𝑞 − 𝜖, 0
28
Nearest Vertex Approach
Normalized
inequality
constraints
Normalized
equality
constraints
Tolerance