Cisco Data Center: Enabling PBR (Policy Based Routing) On Cisco Nexus 9Ks

I've recently came upon the need to do some PBR (Policy Based Routing) on some core Nexus 9Ks within a data center environment.  Its interesting, to say the least, that Cisco is full of "We don't support" statements.  It makes me miss the Brocade days, even though they were not perfect either (although better).  So before we get into the config part, lets take a look at the "Cisco doesn't support" statements that they make:

1.  A policy-based routing route map can have only one match or set statement per route-map statement.
2.  A match command cannot refer to more than one ACL in a route map used for policy-based routing.
3.  The same route map can be shared among different interfaces for policy-based routing as long as the interfaces belong to the same virtual routing and forwarding (VRF) instance.
4.  Using a prefix list as a match criteria is not supported. Do not use a prefix list in a policy-based routing route-map.
5.  Policy-based routing supports only unicast traffic. Multicast traffic is not supported.
6.  Policy-based routing is not supported with inbound traffic on FEX ports.
7.  Policy-based routing is not supported with Layer 3 port-channel subinterfaces.
8.  An ACL used in a policy-based routing route map cannot include deny access control entries (ACEs).
9.  Policy-based routing is supported only in the default system routing mode.
10.  The Cisco Nexus 9000 Series switches do not support the set vrf and set default next-hop commands.
11.  Policy-based routing traffic cannot be balanced if the next hop is recursive over ECMP paths. Instead, use the set {ip | ipv6} next-hop ip-address load-share command to specify the adjacent next hops.
12.  Beginning with Cisco NX-OS Release 6.1(2)I3(2), the Cisco Nexus 9000 Series switches support policy-based ACLs (PBACLs), also referred to as object-group ACLs. For more information, see the Cisco Nexus 9000 Series NX-OS Security Configuration Guide.
13.  If you are familiar with the Cisco IOS CLI, be aware that the Cisco NX-OS commands for this feature might differ from the Cisco IOS commands that you would use.

Now, lets get into the config of this.  I have two Nexus 9Ks as my core.  I'm using L3 ports in this particular case.
CORE2(config)# feature pbr
CORE2(config)#  ip access-list PBR_2_9504s_PERMIT
CORE2(config-acl)# permit ip 10.45.0.0/16 any
CORE2(config-acl)# exit
CORE2(config)# ip access-list PBR_2_9504s_DENY
CORE2(config-acl)# permit ip 10.45.0.0/16 10.0.0.0/8
CORE2(config-acl)# exit
CORE2(config)# route-map PBR_2_9504s deny 10
CORE2(config-route-map)# match ip address PBR_2_9504s_DENY
CORE2(config-route-map)# route-map PBR_2_9504s permit 20
CORE2(config-route-map)# match ip address PBR_2_9504s_PERMIT
CORE2(config-route-map)# set ip next-hop 192.168.10.1 192.168.10.2 load-share

Now, lets apply it to the two L3 interfaces:
int eth 1/1
ip policy route-map PBR_2_9504s
int eth 2/1
ip policy route-map PBR_2_9504s

Lets look at the config for a moment.  Notice that the route-map references two statements.  "route-map PBR_2_9504s deny 10" points to ACL "PBR_2_9504s_DENY" in the statement "match ip address PBR_2_9504s_DENY". That is because of this particular "Cisco doesnt support" statements:
An ACL used in a policy-based routing route map cannot include deny access control entries (ACEs).
Then you move on to the permit statements, of which you want to permit the particular action.  In this case, I want to set the next hop to two different IPs (because of the redundancy in the network).  "route-map PBR_2_9504s permit 20" gets me to the permit actions.  It points to the ACL of "PBR_2_9504s_PERMIT" in the command "match ip address PBR_2_9504s_PERMIT".  Next, I send it to the next hops with the load sharing command "set ip next-hop 192.168.10.1 192.168.10.2 load-share".  I then apply it to the interfaces.

Now, some other things to know about the Nexus 9K and PBR.  You have to have Enterprise Services licensing.
CORE2# sh license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
TP_SERVICES_PKG               No    -   Unused             -
NETWORK_SERVICES_PKG          Yes   -   Unused Never       -
LAN_ENTERPRISE_SERVICES_PKG   Yes   -   In use Never       -
--------------------------------------------------------------------------------
CORE2#

Next, the load-sharing is per flow, not per packet.  This is per Cisco documentation:
"You can optionally configure this command for next-hop addresses to load balance traffic for up
to 32 IP addresses. In this case, Cisco NX-OS sends all traffic for each IP flow to a particular IP
next-hop address."

Also, if you use the "set" command in the first part of the route-map, it will have no effect.  Cisco says this:
"The set command has no effect inside a route-map... deny statement."