I've first brought back my most popular posts.

I hope to take time to write new ones.

The platforms

For those who are interested, this time the blog is running on Ghost behind an NGINX.

My first intent was to use Ghost as headless CMS but as it is not my job to write frontend (AKA I'm too lazy) and Ghost built-in front-end was more thant OK, I've decided to "re"start with Ghost only...

NGINX is used as front-end to ensure some security features like:

• SSL layer, it is responsible to redirect clients to https URLs and to renew the Let's Encrypt certificate
• It protects the admin URLs of  Ghost back-end with client certificate authentication
• Redirect error pages

Ghost is mainly used as back-end but with the integrated front-end features.

• From day-1, very few tuning as been done
• In the future, I might work on a different theme but it is certainly not the main goal... I'm a Network Engineer before all... ;)

Learnt from past...

Everything is running on virtualized environments

• NGINX is running on a qemu VM dedicated to security. All the config is backed up on my home NAS
• Ghost is running on a private multi-purpose Docker host and content is backed up to my home NAS

Security

If you struggle with SELinux but you don't want to simply disable it, here is a good start

Modifying SELinux Settings for Full NGINX and NGINX Plus Functionality

When Security-Enhanced Linux (SELinux) is enabled for Red Hat Enterprise Linux (RHEL) and related distros, its default settings prevent NGINX and NGINX Plus from performing some operations. This article explains how to modify SELinux settings to permit full functionality.

NGINXOwen Garrett of F5

Here is a post that shows how easy it is to use client-side certificate authentication with NGINX. I used it to protect the admin pages of this blog... It took me less than 15 minutes to set it up...

Client-Side Certificate Authentication with nginx

Authentication in applications is tough. If you decide to roll your own, security issues are nearly guaranteed. Most anyone who writes software for a living will tell you to use something you didn’t write; that’s battle-tested and in wide use. Open source is even better; hopefully that many eyes and…

Nathan Wittstock

System

Some interesting info about systemd services...

How to create systemd service unit in Linux - LinuxConfig.org

In this tutorial we analyze the structure of systemd ”.service” units, and examine the most commonoptions which can be used to modify how the service behaves. We see how to set dependencies for a service and how can wespecify the commands to be executed when it is started stopped or reloaded. The …

Linux Tutorials - Learn Linux ConfigurationEgidio Docile

RPM probes can be icmp, tcp, udp or HTTP based. The lasts three require the target to be configured to respond to the probes. The probes can measure packet loss, jitter, egress or ingress jitter, RTT, egress or ingress delay, deviation of the RTT or deviate of egress or ingress delay. Based on those measures, the RPM test can be marked successful or failed.

RPM configuration

Here is an exemple of RPM test using icmp-ping. The test consists in 3 icmp-ping sent to 12.12.12.12 with 1 second interval between each ping and then it start again after an interval of 5 seconds. The test is failed if the three pings are not answered.

[edit services]
rpm {
    probe my-rpm {
        test my-test {
            probe-type icmp-ping;
            target address 12.12.12.12;
            probe-count 3;
            probe-interval 1;
            test-interval 5;
            thresholds {
                successive-loss 3;
            }
        }
    }
}

You can check the probe results as follow:

root@srx11> show services rpm probe-results
    Owner: my-rpm, Test: my-test
    Target address: 12.12.12.12, Probe type: icmp-ping, Test size: 3 probes
    Probe results:
      Response received, Mon Dec 30 22:27:20 2013, No hardware timestamps
      Rtt: 1708 usec
    Results over current test:
      Probes sent: 2, Probes received: 2, Loss percentage: 0
      Measurement: Round trip time
        Samples: 2, Minimum: 1707 usec, Maximum: 1708 usec, Average: 1708 usec,
        Peak to peak: 1 usec, Stddev: 0 usec, Sum: 3415 usec
    Results over last test:
      Probes sent: 3, Probes received: 3, Loss percentage: 0
      Test completed on Mon Dec 30 22:27:12 2013
      Measurement: Round trip time
        Samples: 3, Minimum: 1720 usec, Maximum: 1855 usec, Average: 1776 usec,
        Peak to peak: 135 usec, Stddev: 57 usec, Sum: 5329 usec
    Results over all tests:
      Probes sent: 3305, Probes received: 3278, Loss percentage: 0
      Measurement: Round trip time
        Samples: 3278, Minimum: 1482 usec, Maximum: 22378 usec,
        Average: 1928 usec, Peak to peak: 20896 usec, Stddev: 495 usec,
        Sum: 6318709 usec

You can see the current ongoing test results followed by the last test completed results followed by the consolidated results of all the past tests.

You may also see the history of the last 50 tests.

root@srx11> show services rpm history-results
    Owner, Test                 Probe received              Round trip time
    my-rpm, my-test             Mon Dec 30 22:28:27 2013             1957 usec
    my-rpm, my-test             Mon Dec 30 22:28:30 2013             1799 usec
    my-rpm, my-test             Mon Dec 30 22:28:35 2013             1881 usec
    my-rpm, my-test             Mon Dec 30 22:28:38 2013             1887 usec
    my-rpm, my-test             Mon Dec 30 22:28:41 2013             1988 usec
    my-rpm, my-test             Mon Dec 30 22:28:46 2013             1844 used
(...)

RPM usage

RPM results can be used to influence the routing by injecting a static route into the routing table if the RPM probe fails. This is done with an ip-monitoring policy as follow.

[edit services]
ip-monitoring {
    policy default-route {
        match {
            rpm-probe my-rpm;
        }
        then {
            preferred-route {
                route 0.0.0.0/0 {
                    next-hop 10.11.21.21;
                }
            }
        }
    }
}

As long as the probe is successful, the default route is the static route configured under routing-option (next-hop 10.11.12.12).

root@srx11> show services ip-monitoring status

Policy - default-route (Status: PASS)
  RPM Probes:
    Probe name             Test Name       Address          Status
    ---------------------- --------------- ---------------- ---------
    my-rpm                 my-test         12.12.12.12      PASS
  Route-Action:
    route-instance    route             next-hop         state
    ----------------- ----------------- ---------------- -------------
    inet.0            0.0.0.0/0         10.11.21.21      NOT-APPLIED

root@srx11> show route 0/0 exact

inet.0: 23 destinations, 23 routes (19 active, 0 holddown, 4 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 04:29:32
                    > to 10.11.12.12 via fe-0/0/7.1112

As soon as the probes fail, a new default route is injected into the routing table (next-hop 10.11.21.21)

root@srx11> show services ip-monitoring status

Policy - default-route (Status: FAIL)
  RPM Probes:
    Probe name             Test Name       Address          Status
    ---------------------- --------------- ---------------- ---------
    my-rpm                 my-test         12.12.12.12      FAIL
  Route-Action:
    route-instance    route             next-hop         state
    ----------------- ----------------- ---------------- -------------
    inet.0            0.0.0.0/0         10.11.21.21      APPLIED

root@srx11> show route 0/0 exact

inet.0: 23 destinations, 24 routes (19 active, 0 holddown, 4 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/1] 00:00:47, metric2 0
                    > to 10.11.21.21 via fe-0/0/3.1121
                    [Static/5] 04:32:36
                    > to 10.11.12.12 via fe-0/0/7.1112

That’s all folks!

axsl01/Admin# sh ft group status
FT Group                     : 1
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Id                      : 1
No. of Contexts              : 1
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync status      : Startup configuration sync has completed

FT Group                     : 2
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
Peer State                   : FSM_FT_STATE_STANDBY_COLD
Peer Id                      : 1
No. of Contexts              : 1
Running cfg sync status      : Peer in Cold State. Incremental Sync Failure: script file not configured

Startup cfg sync status      : Peer in Cold State. Incremental Sync Failure: script file not configured

First step is, of course, to copy the missing files to the standby ACE. Script files have to be copied via tftp, certificates and keys may be copied via tftp or terminal copy/paste. In newer firmware releases you can easily identified what caused the ACE to go into that cold state:

axsl02/C1# sh ft config-error
Mon Aug 13 09:19:18 UTC 2012

`script file 2 test.tcl`
Error: Unable to locate the script file 'test.tcl'

Once the files are copied, you have to bring back the standby ACE to the standby hot state. As of now, I know four ways to do that. Actually only three are working, the fourth one (provided by Cisco) never worked for me.

Hard way

1. Reboot the standby ACE.

axsl01/Admin# reload
This command will reboot the system
Save configurations for all the contexts. Save? [yes/no]: [yes] yes
Perform system reload. [yes/no]: [yes] yes

In this case, you have to ensure all the contexts are active or at least in standby hot state on the other ACE. It takes a “very” long time (ACE boot time is +/- 10 minutes) before you can validate it worked.

Quick and dirty way – aka I’m feeling lucky today

1. Go to the Admin context on the active ACE.
2. Quickly deactivate/activate the ft group of the context you want to fix.

axsl01/Admin# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
axsl01/Admin(config)# ft group 2
axsl01/Admin(config-ft-group)# no inservice
axsl01/Admin(config-ft-group)# inservice
axsl01/Admin(config-ft-group)# end
axsl01/Admin# sh ft group 2 status

FT Group                     : 2
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
Peer State                   : FSM_FT_STATE_STANDBY_BULK
Peer Id                      : 1
No. of Contexts              : 1
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync status      : Startup configuration sync has completed
axsl01/Admin# sh ft group 2 status             (after some time)

FT Group                     : 2
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Id                      : 1
No. of Contexts              : 1
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync status      : Startup configuration sync has completed

If you’re quick enough, the active ACE will keep serving the active sessions, only the new sessions will be refused during the period of time the context is not in service, few ms if you’re quick enough.

This method may still be unacceptable in some case and might be catastrophic if you (or your terminal client) are not quick enough.

Cautious way – aka exp-Networks way

1. Make the Admin context active on the standby ACE, perform a switchover of the Admin context if required.
2. Disable the running-config synchronization for the Admin context.
3. Deactivate the ft group of the context you want to fix. Since you’ve disabled the running-config synchronization, the command only take effect locally, in other words the active context on the other ACE remains active.
4. Activate the ft group again and wait until the context stabilize in standby hot state.
5. Enable the running-config synchronization for the Admin context again.
6. Perform a switchover of the Admin context if required.

axsl02/Admin# sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_STANDBY_HOT       
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: Admin
                Context Id: 0   Running Cfg Sync Status: Successful
                
FT Group ID: 2  My State:FSM_FT_STATE_STANDBY_COLD
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: C1
                Context Id: 1   Running Cfg Sync Status: Successful

axsl02/Admin# ft switchover 1
This command will cause card to switchover (yes/no)?  [no] yes
axsl02/Admin#

NOTE: Configuration mode is enabled on all sessions
NOTE: Configuration mode has been disabled on all sessions
NOTE: Configuration mode is enabled on all sessions

axsl02/Admin# sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: Admin
                Context Id: 0   Running Cfg Sync Status: Successful

FT Group ID: 2  My State:FSM_FT_STATE_STANDBY_COLD
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: C1
                Context Id: 1   Running Cfg Sync Status: Successful

axsl02/Admin# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
axsl02/Admin(config)# no ft auto-sync running-config
axsl02/Admin(config)# ft group 2
axsl02/Admin(config-ft-group)# no inservice
axsl02/Admin(config-ft-group)# do sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: Admin
                Context Id: 0   Running Cfg Sync Status: Successful

FT Group ID: 2  My State:FSM_FT_STATE_INIT
                Peer State:FSM_FT_STATE_UNKNOWN
                Context Name: C1
                Context Id: 1   Running Cfg Sync Status: Successful

We can check on the other ACE that context C1 is still active.

axsl01/Admin# sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_STANDBY_HOT
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: Admin
                Context Id: 0   Running Cfg Sync Status: Successful

FT Group ID: 2  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_INIT
                Context Name: C1
                Context Id: 1   Running Cfg Sync Status: Successful

Back on the other ACE to bring the context back into service.

axsl02/Admin(config-ft-group)# inservice
axsl02/Admin(config-ft-group)# ft auto-sync running-config
axsl02/Admin(config)#

NOTE: Configuration mode has been disabled on all sessions
NOTE: Configuration mode is enabled on all sessions

axsl02/Admin(config)# do sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: Admin
                Context Id: 0   Running Cfg Sync Status: Successful

FT Group ID: 2  My State:FSM_FT_STATE_STANDBY_HOT
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: C1
                Context Id: 1   Running Cfg Sync Status: Successful

axsl02/Admin(config)# end
axsl02/Admin# ft switchover 1
This command will cause card to switchover (yes/no)?  [no] yes

NOTE: Configuration mode has been disabled on all sessions

Cisco way

1. Go to the context you want to fix on the active ACE, disable running-config and startup-config synchronization
2. Then enable them back.

Every time I’ve tried this, there has been indeed a resynchronization but the standby ACE ended back to standby cold state.

axsl01/C1# sh ft group brief

FT Group ID: 2  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_COLD
                Context Name: C1
                Context Id: 1   Running Cfg Sync Status: Successful

axsl01/C1(config)# no ft auto-sync running-config
axsl01/C1(config)# no ft auto-sync startup-config
axsl01/C1(config)# ft auto-sync startup-config
axsl01/C1(config)# ft auto-sync running-config

NOTE: Configuration mode has been disabled on all sessions
NOTE: Configuration mode is enabled on all sessions

axsl01/C1(config)# do sh ft group brief

FT Group ID: 2  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_COLD
                Context Name: C1
                Context Id: 1   Running Cfg Sync Status: Successful

That’s all folks! Leave a comment if you liked, disliked, have additional inputs, queries, want to hire our services, want just say hello…

Firewalls J1 and J2 are in cluster, only one device is actively forwarding traffic. The eBGP peering will be done with the active device via a virtual IP called VSI in ScreenOS jargon. The cluster is in AS 64999. As the active device is J1 the traffic will be forced through C1 which is connected to the same switch.

C1 and C2 are core routers running BGP in AS 65000, they have an iBGP session between them and will form an eBGP peering with the cluster’s VSI.

Cisco IOS configuration

Cisco routers have a very basic BGP configuration.

C1

interface Vlan644
 ip address 10.6.44.1 255.255.255.248

interface Vlan645
 ip address 10.6.45.1 255.255.255.248

router bgp 65000
 neighbor 10.6.45.2 remote-as 65000
 neighbor 10.6.45.2 password exp-networks
 neighbor 10.6.44.4 remote-as 64999
 neighbor 10.6.44.4 password exp-networks
 network 10.6.45.0 mask 255.255.255.248

C2

interface Vlan644
 ip address 10.6.44.2 255.255.255.248

interface Vlan645
 ip address 10.6.45.2 255.255.255.248

router bgp 65000
 neighbor 10.6.45.1 remote-as 65000
 neighbor 10.6.44.1 password exp-networks
 neighbor 10.6.44.4 remote-as 64999
 neighbor 10.6.44.4 password exp-networks
 network 10.6.45.0 mask 255.255.255.248

Juniper ScreenOS configuration

Only the active ScreenOS device needs to be configured, the configuration will be replicated to the standby device thanks to NSRP. For demonstration purpose the cluster will send the default route to the routers and refuse it from the routers… As the active node is connected to C1 the traffic will be forced to go through that router with the help of MED and weight.

J1

set interface "aggregate1.644" tag 644 zone "Trust"
set interface aggregate1.644 ip 10.6.44.4/29
set interface aggregate1.644 route
set interface aggregate1.644 manage-ip 10.6.44.5
set interface aggregate1.644 protocol bgp

set vrouter "trust-vr"
set protocol bgp 64999
set enable
set neighbor 10.6.44.1 remote-as 65000
set neighbor 10.6.44.1 enable
set neighbor 10.6.44.1 md5-authentication exp-networks
set neighbor 10.6.44.2 remote-as 65000
set neighbor 10.6.44.2 enable
set neighbor 10.6.44.2 md5-authentication exp-networks
set ipv4 neighbor 10.6.44.1 activate
set ipv4 neighbor 10.6.44.1 med 10
set ipv4 neighbor 10.6.44.1 advertise-def-route
set ipv4 neighbor 10.6.44.1 reject-default-route
set ipv4 neighbor 10.6.44.1 weight 200
set ipv4 neighbor 10.6.44.2 activate
set ipv4 neighbor 10.6.44.2 med 20
set ipv4 neighbor 10.6.44.2 advertise-def-route
set ipv4 neighbor 10.6.44.2 reject-default-route
exit
exit

J2

To be complete, J2 is configured with its own manage-ip.

set interface aggregate1.644 manage-ip 10.6.44.6

Verifications

Few second after the configuration the BGP peering should be established.

Check BGP peering on ScreenOS

On Juniper devices

J1(M)-> get vrouter trust-vr protocol bgp neighbor
Peer AS Remote IP   Local IP      Wt Status   State     ConnID Up/Down
-----------------------------------------------------------------------
  65000 10.6.44.1   10.6.44.4    100 Enabled  ESTABLISH   1020 00:18:18
  65000 10.6.44.2   10.6.44.4    100 Enabled  ESTABLISH   1043 00:00:19

total 2 BGP peers shown

On Cisco routers

C1# sh ip bgp summary
BGP router identifier 10.6.45.1, local AS number 65000
BGP table version is 96, main routing table version 96
2 network entries using 274 bytes of memory
3 path entries using 204 bytes of memory
7/3 BGP path/bestpath attribute entries using 980 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1530 total bytes of memory
BGP activity 15/9 prefixes, 51/44 paths, scan interval 15 secs

Neighbor   V    AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.6.44.4  4 64999    5964    7607     96   0    0 00:35:02        1
10.6.45.2  4 65000     974     925     96   0    0 00:35:06        1

Check the bgp table

On Juniper devices

J1(M)-> get vrouter trust-vr protocol bgp rib-in
i: IBGP route, e: EBGP route, >: best route, *: valid route
               Prefix         Nexthop    Wt  Pref   Med Orig    AS-Path
-----------------------------------------------------------------------
Total ipv4 routes in rib-in: 2 (0 in flap-damping history)
-----------------------------------------------------------------------
>e*      10.6.45.0/29       10.6.44.1   200   100     0  IGP   65000
 e       10.6.45.0/29       10.6.44.2   100   100     0  IGP   65000
Total no. of ipv4 entries shown: 2

Or to see all the details

J1(M)-> get vrouter trust-vr protocol bgp rib-in 10.6.45.0/29
Prefix: 10.6.45.0/29
Nexthop: 10.6.44.1, Weight: 200, Local Pref: 100, MED: 0, Flags: 0x486 0x88, Orig: IGP
AS segment type: AS_SEQ, AS path:65000

Prefix: 10.6.45.0/29
Nexthop: 10.6.44.2, Weight: 100, Local Pref: 100, MED: 0, Flags: 0x404 0x88, Orig: IGP
AS segment type: AS_SEQ, AS path:65000

On Cisco devices. C1 should only see the default route from J1 while C2 should see it from C1 and J1 and select the one through C1 due to the better MED.

C1#sh ip bgp 
BGP table version is 96, local router ID is 10.10.254.164
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 0.0.0.0          10.6.44.4               10             0 64999 i
* i10.6.45.0/29     10.6.45.2                0    100      0 i
*>                  0.0.0.0                  0         32768 i

C2#sh ip bgp
BGP table version is 152, local router ID is 10.10.254.201
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*>i0.0.0.0          10.6.45.1               10    100      0 64999 i
*                   10.6.44.4               20             0 64999 i
*> 10.6.45.0/29     0.0.0.0                  0         32768 i
* i                 10.6.45.1                0    100      0 i

Or to see all the details (on C2 only)

C2# sh ip bgp 0.0.0.0/0
BGP routing table entry for 0.0.0.0/0, version 148
Paths: (2 available, best #1)
  Advertised to update-groups:
     2
  64999
    10.6.45.1 from 10.6.45.1 (10.6.45.1)
      Origin IGP, metric 10, localpref 100, valid, internal, best
  64999
    10.6.44.4 from 10.6.44.4 (10.6.44.4)
      Origin IGP, metric 20, localpref 100, valid, external

That’s it… Not very complex and it may ease your life a lot.

Now, what happen when the content is dynamic? The best example is a shopping cart on an e-commerce website, you browse a catalog and add stuffs you want to purchase. At any time you may consult the content of your shopping cart. Usually, the shopping cart is created on one and only one application server. So now, even if we have a server farm of four servers, every request related to the shopping cart (e.g. add or remove items, view the content) must go to the same application server. If the request goes to another application server, this one won’t be able to treat the request because it has no knowledge about that shopping cart instance…

To tackle that problem, the ACE use a technique called stickiness. A particular client is stick to a specific server. The ACE allows several methods to track the association client/server.

• source IP
• destination IP
• source and destination IP
• HTTP Cookie
• HTTP Header
• HTTP Content
• Generic layer 4 parsing
• RADIUS
• SIP

Let’s examine the most common ones based on the following starting config.

reserver host A
  ip address 10.10.10.1
  inservice
reserver host B
  ip address 10.10.10.2
  inservice

serverfarm host AB
  rserver A 80
    inservice
  rserver B 80
    inservice

class-map match-all VIP-80
  match virtual-address 100.100.100.1 tcp eq www

policy-map type loadbalance first-match LB-80
  class class-default
    serverfarm AB

policy-map multi-match VIP
  class VIP-80
    loadbalance vip inservice
    load balance policy LB-80

interface vlan100
  service-policy input VIP

With this configuration, the ACE will load-balance all the requests in a round-robin fashion between rservers A and B.

Source IP based stickiness

This method well suited only if the ACE sees enough different source IP. If all your traffic goes through a reverse-proxy this method is not for you since the ACE will see all the clients coming from the same IP and will therefore send everybody to the same application server.

On the other hand, the more different source IP the ACE sees, the more memory it will use to keep track of all the sticky sessions. If memory usage becomes an issue, you may chose a smaller netmask or go for another method.

sticky ip-netmask 255.255.255.255 address source sticky-ip
 replicate sticky
 serverfarm AB
 
 class-map match-all VIP-81
   match virtual-address 100.100.100.1 tcp eq 81

 policy-map type loadbalance first-match LB-81
   class class-default
     sticky-serverfarm sticky-ip
     
  policy-map multi-match VIP
   class VIP-81
     loadbalance vip inservice
     load balance policy LB-81

To check to which real server a client is associated, just issue the command “show sticky database type ip-netmask”.

ACE# show sticky database type ip-netmask source
sticky group : sticky-ip
type         : IP    
timeout      : 1440          timeout-activeconns : FALSE
  sticky-entry          rserver-instance                 time-to-expire flags   
  ---------------------+--------------------------------+--------------+-------+
  175735071             A:80                             85962          - 
sticky group : sticky-ip
type         : IP    
timeout      : 1440          timeout-activeconns : FALSE
  sticky-entry          rserver-instance                 time-to-expire flags   
  ---------------------+--------------------------------+--------------+-------+
  175669553             B:80                             86373          - 

The numbers under sticky-entry are the source IP converted into decimal.

HTTP Cookie based stickiness

In the Web world the usage of cookies is well known. The ACE can use them or even generate its own cookie to achieve session stickiness. In the example here below, we will make the ACE to generate its own cookie. The main advantage of that method is the low memory consumption as the ACE built the cookie-rserver mapping in advance. Only one entry per real server is required. With cookies generated by the application, the ACE needs to create a dynamic entry per cookie value and, therefore, might consume more memory.

sticky http-cookie myCookie sticky-cookie
 cookie insert browser-expire
 serverfarm AB
  
  class-map match-all VIP-82
    match virtual-address 100.100.100.1 tcp eq 82
 
  policy-map type loadbalance first-match LB-82
    class class-default
      sticky-serverfarm sticky-cookie
      
  policy-map multi-match VIP
    class VIP-82
      loadbalance vip inservice
      load balance policy LB-82

As our serverfarm AB contains only two rservers, the ACE will generate only two cookie values based on serverfarm name, rserver name and rserver port. To check which value is assiciated to which rserver, issue the command ‘show sticky cookie-insert group’.

ACE# show sticky cookie-insert group sticky-cookie
     Cookie   |        HashKey       |           rserver-instance   
  ------------+----------------------+----------------------------------------+
  R7072362623 | 15959351102845316184 | AB/A:80
  R1637696807 | 14926835405428408087 | AB/B:80

HTTP Header base stickiness

Instead of using a cookie, you may want to use application specific HTTP header (e.g. MSISDN in the mobile world). The usage is almost the same as with the application generated cookie.

sticky http-header myHeader sticky-header
 replicate sticky
 serverfarm AB
  
  class-map match-all VIP-83
    match virtual-address 100.100.100.1 tcp eq 83
 
  policy-map type loadbalance first-match LB-83
    class class-default
      sticky-serverfarm sticky-header
      
  policy-map multi-match VIP
    class VIP-83
      loadbalance vip inservice
      load balance policy LB-83

ACE# show sticky database http-header myHeader

Now, in cluster mode, you can do the software upgrade with no or very limited impact if you follow the correct sequence of operations. Here are the steps I used last time and it went perfectly and transparent for the users.

Note this procedure has been tested on ACE modules for Catalyst 6500 only but it should stay valid for the ACE 4710 appliances.

Step 1

First you need to make sure all the contexts are properly synchronized and the standby contexts are in STANDBY_HOT state.

ACE_1/Admin# sh ft group brief
FT Group ID: 1  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: Admin     Context Id: 0
FT Group ID: 2  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_COLD
                Context Name: C1        Context Id: 4
FT Group ID: 3  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: C2        Context Id: 3

Here as you can see context C1 is stuck in STANDBY_COLD state. Usually put that context out of service on the standby ACE and then put it back in service solve the issue. If it is not the case you won’t have a fully transparent software upgrade for that context; current session will be dropped but new session will be accepted after the failover. If it is acceptable for you, go on with the upgrade otherwise try to find out why it is not in STANDBY_HOT state.

Note it might take several minutes to leave the STANDBY_BULK state (it took 2 minutes during my tests).

ACE_2/Admin(config)# ft group 2
ACE_2/Admin(config-ft-group)# no inservice
ACE_2/Admin(config-ft-group)# do sh ft group 2 detail

FT Group                     : 2
No. of Contexts              : 1
Context Name                 : C1
Context Id                   : 4
Configured Status            : out-of-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_INIT
My Config Priority           : 90
My Net Priority              : 90
My Preempt                   : Enabled
Peer State                   : FSM_FT_STATE_UNKNOWN
Peer Config Priority         : Unknown
Peer Net Priority            : Unknown
Peer Preempt                 : Unknown
Peer Id                      : 1
Last State Change time       : Wed Feb  3 14:35:36 2010
Running cfg sync enabled     : Enabled
Running cfg sync status      :
Startup cfg sync enabled     : Enabled
Startup cfg sync status      :
Bulk sync done for ARP: 0
Bulk sync done for LB: 0
Bulk sync done for ICM: 0
ACE_2/Admin(config-ft-group)# inservice

NOTE: Configuration mode has been disabled on all sessions

ACE_2/Admin(config-ft-group)# do sh ft group 2 detail

FT Group                     : 2
No. of Contexts              : 1
Context Name                 : C1
Context Id                   : 4
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_STANDBY_BULK
My Config Priority           : 90
My Net Priority              : 90
My Preempt                   : Enabled
Peer State                   : FSM_FT_STATE_ACTIVE
Peer Config Priority         : 120
Peer Net Priority            : 120
Peer Preempt                 : Enabled
Peer Id                      : 1
Last State Change time       : Wed Feb  3 14:36:02 2010
Running cfg sync enabled     : Enabled
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync enabled     : Enabled
Startup cfg sync status      : Startup configuration sync has completed
Bulk sync done for ARP: 1
Bulk sync done for LB: 0
Bulk sync done for ICM: 0
ACE_2/Admin(config-ft-group)# do sh ft group 1 detail

FT Group                     : 2
No. of Contexts              : 1
Context Name                 : C1
Context Id                   : 4
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_STANDBY_HOT
My Config Priority           : 90
My Net Priority              : 90
My Preempt                   : Enabled
Peer State                   : FSM_FT_STATE_ACTIVE
Peer Config Priority         : 120
Peer Net Priority            : 120
Peer Preempt                 : Enabled
Peer Id                      : 1
Last State Change time       : Wed Feb  3 14:37:51 2010
Running cfg sync enabled     : Enabled
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync enabled     : Enabled
Startup cfg sync status      : Startup configuration sync has completed
Bulk sync done for ARP: 1
Bulk sync done for LB: 2
Bulk sync done for ICM: 2

Step 2

On the ACE, preemption is enabled by default for all the  contexts. It needs to be disabled to perform a manual failover.

ACE_1/Admin(config)# ft group 1
ACE_1/Admin(config-ft-group)# no preempt
ACE_1/Admin(config-ft-group)# ft group 2
ACE_1/Admin(config-ft-group)# no preempt
ACE_1/Admin(config-ft-group)# ft group 3
ACE_1/Admin(config-ft-group)# no preempt
ACE_1/Admin(config-ft-group)# end

Step 3

Download the new software image to the active and standby ACEs. Here I’ve chosen to use tftp because I hadn’t a ftp server configured in the lab… ftp can be used and is definitely faster.

ACE_1/Admin# copy tftp: image:
Enter source filename[]? c6ace-t1k9-mz.A2_2_3.bin
Enter the destination filename[]? [c6ace-t1k9-mz.A2_2_3.bin]
Address of remote host[]? 10.1.1.1
Trying to connect to tftp server......
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(…)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
TFTP get operation was successful
 31361516 bytes copied
ACE_1/Admin#
ACE_1/Admin# dir image:
 30788103  Apr 15 13:14:48 2009 c6ace-t1k9-mz.A2_1_4a.bin
 31361516  Feb  3 14:43:45 2010 c6ace-t1k9-mz.A2_2_3.bin
          Usage for image: filesystem
          461848576 bytes total used
          577126400 bytes free
         1038974976 total bytes

Check the file size is correct…

Step 4

Change the boot string on the active ACE, it will be synced to the standby ACE. By the way, configuration mode is disabled on the standby ACE therefore it is the only option…

ACE_1/Admin# sh run | i boot
Generating configuration....
boot system image:c6ace-t1k9-mz.A2_1_4a.bin
ACE_1/Admin# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
ACE_1/Admin(config)# no boot system image:c6ace-t1k9-mz.A2_1_4a.bin
ACE_1/Admin(config)# boot system image:c6ace-t1k9-mz.A2_2_3.bin
ACE_1/Admin(config)# exit
ACE_1/Admin# wr mem all
Generating configuration....
running config of context Admin saved
Generating configuration....
running config of context C2 saved
Generating configuration....
running config of context C1 saved
Please wait ... sync to compact flash in progress.

This may take a few minutes to complete

Sync Done

Step 5 (optional)

Create checkpoint in all contexts on active and standby devices

ACE_2/Admin# checkpoint create 20100203
Generating configuration....
Created configuration checkpoint '20100203'
ACE_2/Admin# changeto C2

NOTE: Configuration mode has been disabled on all sessions

ACE_2/C2# checkpoint create 20100203
Generating configuration....
Created configuration checkpoint '20100203'
ACE_2/C2# changeto C1

NOTE: Configuration mode has been disabled on all sessions

ACE_2/C1# checkpoint create 20100203
Generating configuration....
Created configuration checkpoint '20100203'
ACE_2/C1# changeto Admin

Step 6

Reload the standby device

ACE_2/Admin# reload
This command will reboot the system
Save configurations for all the contexts. Save? [yes/no]: [yes] no 
(already done in step 4)
Perform system reload. [yes/no]: [yes]

NOTE: Configuration mode is enabled on all sessions

Connection to ACE_2 closed by remote host.
Connection to ACE_2 closed.

Step 7

Check the standby device is running the new software version.

ACE_2/Admin# sh ver
Cisco Application Control Software (ACSW)
TAC support: http://C2 .cisco.com/tac
Copyright (c) 2002-2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://C2 .gnu.org/licenses/gpl.html.

Software
  loader:    Version 12.2[120]
  system:    Version A2(2.3) [build 3.0(0)A2(2.3)]
  system image file: [LCP] disk0:c6ace-t1k9-mz.A2_2_3.bin
  installed license: ACE-VIRT-020

Hardware
  Cisco ACE (slot: 6)
  cpu info:
    number of cpu(s): 2
    cpu type: SiByte
    cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz
    cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz
  memory info:
    total: 827128 kB, free: 256000 kB
    shared: 0 kB, buffers: 1824 kB, cached 0 kB
  cf info:
    filesystem: /dev/cf
    total: 1014624 kB, used: 451040 kB, available: 563584 kB

last boot reason:  reload command by Admin
configuration register:  0x1
ACE_2 kernel uptime is 0 days 0 hour 8 minute(s) 45 second(s)

Step 8

Wait until all the contexts on the standby devices stabilize in STANDBY_WARM or STANDBY_HOT state.

ACE_2/Admin# sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_STANDBY_WARM
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: Admin     Context Id: 0
FT Group ID: 2  My State:FSM_FT_STATE_STANDBY_WARM
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: C1        Context Id: 4
FT Group ID: 3  My State:FSM_FT_STATE_STANDBY_WARM
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: C2        Context Id: 3

For your information, here is what Cisco says about STANDBY_WARM state :

In the STANDBY_WARM state, as with the STANDBY_HOT state, configuration mode is disabled on the standby ACE and configuration and state synchronization continues. A failover from the active to the standby based on priorities and preempt can still occur while the standby is in the STANDBY_WARM state. However, while stateful failover is possible for a WARM standby, it is not guaranteed. In general, modules should be allowed to remain in this state only for a short period.

Step 9

Perform a failover from the active ACE to the standby ACE for all the contexts.

ACE_1/Admin# ft switchover all
This command will cause card to switchover (yes/no)?  [no] yes

NOTE: Configuration mode has been disabled on all sessions

Step 10

Check the newly upgraded ACE is well become active.

ACE_1/Admin# sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_STANDBY_BULK
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: Admin     Context Id: 0
FT Group ID: 2  My State:FSM_FT_STATE_STANDBY_BULK
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: C1        Context Id: 4
FT Group ID: 3  My State:FSM_FT_STATE_STANDBY_BULK
                Peer State:FSM_FT_STATE_ACTIVE
                Context Name: C2        Context Id: 3

Step 11

Reload the 2^nd ACE (previously active).

ACE_1/Admin# reload
This command will reboot the system
Save configurations for all the contexts. Save? [yes/no]: [yes] no
Perform system reload. [yes/no]: [yes]

NOTE: Configuration mode is enabled on all sessions

Connection to ACE_1 closed by remote host.
Connection to ACE_1 closed.

Step 12

When the 2^nd ACE state stabilize to FSM_FT_STATE_STANDBY_HOT state, perform again a failover for all the contexts.

ACE_2/Admin# sh ft group brief

FT Group ID: 1  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: Admin     Context Id: 0
FT Group ID: 2  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: C1        Context Id: 4
FT Group ID: 3  My State:FSM_FT_STATE_ACTIVE
                Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: C2        Context Id: 3

Step 13 (If you’re not superstitious)

Reconfigure preemption if it is in your standard… (personally I don’t like preemption because if a device has failed I prefer to check exactly why before activating it again)

ACE_1/Admin(config)# ft group 1
ACE_1/Admin(config-ft-group)# preempt
ACE_1/Admin(config-ft-group)# ft group 2
ACE_1/Admin(config-ft-group)# preempt
ACE_1/Admin(config-ft-group)# ft group 3
ACE_1/Admin(config-ft-group)# preempt
ACE_1/Admin(config-ft-group)# end
ACE_1/Admin# wr mem

And that’s it, you have upgraded your ACE cluster with no or limited impact. If you find this post helpful you may leave a comment to encourage me to publish more articles…