This is a pretty interesting question, and i'm going to try to contribute with what I can.

First, I think you have to clearly define the boundaries for the game you are trying to create, and define those questions (some may already been answered).

• How far the is the monster aggro ?
• How many monsters at the same time is your target?
• How is your terrain organized? Is it tiled?
• How much collision avoidance do you want?

To quote an answer from here on how pathfinding is done in starcraft 2:

Starcraft II uses a constrained Delaunay triangulation of the map terrain and buildings to produce a navmesh; A* with a funnel filter is used to path along this mesh, taking into account unit radii; then local steering and collision avoidance layers are added on top of that, including a cooperative "push idle units out of the way" feature where it is possible to displace a unit instead of pathing around it in certain cases. Additionally, units moving in parallel are ignored for collision avoidance purposes since they can be guaranteed to not affect each other; [...] SC2 uses six steering forces: following, flocking, grouping, separation, avoidance, and arrival.

So going back to your 3 propositions:

1. NavMesh + RVO -> If there are license issues, then it is not an option. It might be the easiest implementation though.
2. Flow Fields + Physics Engine -> It depends honestly, but it seems really compute intensive in a semi dynamic environment as what you want
3. Steering Behaviors + Physics Engine -> Steering behavior seems to me the way to go. That way you can define some pretty nice flock behaviors for your game, depending for example on the monster type. And it scales well with the mob number. However, i would stay away from physics for collision detection. Simple avoidance behavior is enough, reducing speed or increasing, turning, etc.

Some notes on the Steering behaviors, you can define as much steering forces as you need, with the areas you want. You need at least 3, for flocking/attraction/repulsion, but more are probably interesting.

Then, try to find a library that does steering behaviors to see if it fits your needs (like this? i don't really know any, but it exists).

If the library doesn't fit your needs, then you're up for some fun! But there are enough resources and algorithms to implement flocking behaviors yourself. Example 1 Example 2

If you choose to implement yourself, know there are some nice optimization to be done, as the scope of your agent is reduced for example by its location in the flock. It should anyway be included in some of the algorithms.

Well that's the best of my knowledge, I'm not sure

This is a pretty interesting question, and i'm going to try to contribute with what I can.

First, I think you have to clearly define the boundaries for the game you are trying to create, and define those questions (some may already been answered).

• How far the is the monster aggro ?
• How many monsters at the same time is your target?
• How is your terrain organized? Is it tiled?
• How much collision avoidance do you want?

To quote an answer from here on how pathfinding is done in starcraft 2:

Starcraft II uses a constrained Delaunay triangulation of the map terrain and buildings to produce a navmesh; A* with a funnel filter is used to path along this mesh, taking into account unit radii; then local steering and collision avoidance layers are added on top of that, including a cooperative "push idle units out of the way" feature where it is possible to displace a unit instead of pathing around it in certain cases. Additionally, units moving in parallel are ignored for collision avoidance purposes since they can be guaranteed to not affect each other; [...] SC2 uses six steering forces: following, flocking, grouping, separation, avoidance, and arrival.

So going back to your 3 propositions:

1. NavMesh + RVO -> If there are license issues, then it is not an option. It might be the easiest implementation though.
2. Flow Fields + Physics Engine -> It depends honestly, but it seems really compute intensive in a semi dynamic environment as what you want
3. Steering Behaviors + Physics Engine -> Steering behavior seems to me the way to go. That way you can define some pretty nice flock behaviors for your game, depending for example on the monster type. And it scales well with the mob number. However, i would stay away from physics for collision detection. Simple avoidance behavior is enough, reducing speed or increasing, turning, etc.

Some notes on the Steering behaviors, you can define as much steering forces as you need, with the areas you want. You need at least 3, for flocking/attraction/repulsion, but more are probably interesting.

Then, try to find a library that does steering behaviors to see if it fits your needs (like this? i don't really know any, but it exists).

If the library doesn't fit your needs, then you're up for some fun! But there are enough resources and algorithms to implement flocking behaviors yourself. Example 1 Example 2

If you choose to implement yourself, know there are some nice optimization to be done, as the scope of your agent is reduced for example by its location in the flock. It should anyway be included in some of the algorithms.

Well that's the best of my knowledge, I'm not sure anyway