Service Resiliency

Instead of failing immediately, retry the Service N more times

We will make pod recommendation-v2 fail 100% of the time. Get one of the pod names from your system and replace on the following command accordingly:

You will be inside the application container of your pod recommendation-v2-2036617847-spdrb. Now execute:

This is a special endpoint that will make our application return only `503`s.

You will see it works every time because Istio will retry the recommendation service automatically and it will land on v1 only.

If you open Kiali, you will notice that v2 receives requests, but that failing request is never returned to the user as preference will retry to establish the connection with recommendation, and v1 will reply.

In Kiali, go to Graph, select the recommendation square, and place the mouse over the red sign, like the picture bellow.

Now, make the pod v2 behave well again

You will be inside the application container of your pod recommendation-v2-2036617847-spdrb. Now execute:

The application is back to random load-balancing between v1 and v2

Wait only N seconds before giving up and failing. At this point, no other virtual service nor destination rule (in tutorial namespace) should be in effect. To check it run kubectl get virtualservice kubectl get destinationrule and if so kubectl delete virtualservice virtualservicename -n tutorial and kubectl delete destinationrule destinationrulename -n tutorial

First, introduce some wait time in recommendation v2 by making it a slow performer with a 3 second delay by running the command

Hit the customer endpoint a few times, to see the load-balancing between v1 and v2 but with v2 taking a bit of time to respond

Then add the timeout rule

You will see it return v1 after waiting about 1 second. You don’t see v2 anymore, because the response from v2 expires after the timeout period and it is never returned.