istio-proxy注入后HTTP类型探针的返回码变化

Liveness probe failed: HTTP probe failed with statuscode: 500

     89 // DoHTTPProbe checks if a GET request to the url succeeds.
     90 // If the HTTP response code is successful (i.e. 400 > code >= 200), it returns Success.
     91 // If the HTTP response code is unsuccessful or HTTP communication fails, it returns Failure.
     92 // This is exported because some other packages may want to do direct HTTP probes.
     93 func DoHTTPProbe(url *url.URL, headers http.Header, client GetHTTPInterface) (probe.Result, string, error) {
     94         req, err := http.NewRequest("GET", url.String(), nil)
     95         if err != nil {
     96                 // Convert errors into failures to catch timeouts.
     97                 return probe.Failure, err.Error(), nil
     98         }
                  ...
    131         if res.StatusCode >= http.StatusOK && res.StatusCode < http.StatusBadRequest {
    132                 if res.StatusCode >= http.StatusMultipleChoices { // Redirect
    133                         klog.V(4).Infof("Probe terminated redirects for %s, Response: %v", url.String(), *res)
    134                         return probe.Warning, fmt.Sprintf("Probe terminated redirects, Response body: %v", body), nil
    135                 }
    136                 klog.V(4).Infof("Probe succeeded for %s, Response: %v", url.String(), *res)
    137                 return probe.Success, body, nil
    138         }
    139         klog.V(4).Infof("Probe failed for %s with request headers %v, response body: %v", url.String(), headers, body)
    140         return probe.Failure, fmt.Sprintf("HTTP probe failed with statuscode: %d", res.StatusCode), nil
    141 }

The command approach works with no changes required, but HTTP requests, TCP probes, and gRPC probes require Istio to make changes to the pod configuration.

The health check requests to the liveness-http service are sent by Kubelet. This becomes a problem when mutual TLS is enabled, because the Kubelet does not have an Istio issued certificate. Therefore the health check requests will fail.

TCP probe checks need special handling, because Istio redirects all incoming traffic into the sidecar, and so all TCP ports appear open. The Kubelet simply checks if some process is listening on the specified port, and so the probe will always succeed as long as the sidecar is running.

Istio solves both these problems by rewriting the application PodSpec readiness/liveness probe, so that the probe request is sent to the sidecar agent. For HTTP and gRPC requests, the sidecar agent redirects the request to the application and strips the response body, only returning the response code. For TCP probes, the sidecar agent will then do the port check while avoiding the traffic redirection.

    289 // FormatProberURL returns a set of HTTP URLs that pilot agent will serve to take over Kubernetes
    290 // app probers.
    291 func FormatProberURL(container string) (string, string, string) {
    292         return fmt.Sprintf("/app-health/%v/readyz", container),
    293                 fmt.Sprintf("/app-health/%v/livez", container),
    294                 fmt.Sprintf("/app-health/%v/startupz", container)
    295 }

    590 func applyRewrite(pod *corev1.Pod, req InjectionParameters) error {
    591         valuesStruct := &opconfig.Values{}
    592         if err := gogoprotomarshal.ApplyYAML(req.valuesConfig, valuesStruct); err != nil {
    593                 log.Infof("Failed to parse values config: %v [%v]\n", err, req.valuesConfig)
    594                 return fmt.Errorf("could not parse configuration values: %v", err)
    595         }
    596 
    597         rewrite := ShouldRewriteAppHTTPProbers(pod.Annotations, valuesStruct.GetSidecarInjectorWebhook().GetRewriteAppHTTPProbe())
    598         sidecar := FindSidecar(pod.Spec.Containers)
    599 
    600         // We don't have to escape json encoding here when using golang libraries.
    601         if rewrite && sidecar != nil {
    602                 if prober := DumpAppProbers(&pod.Spec, req.meshConfig.GetDefaultConfig().GetStatusPort()); prober != "" {
    603                         sidecar.Env = append(sidecar.Env, corev1.EnvVar{Name: status.KubeAppProberEnvName, Value: prober})
    604                 }
    605                 patchRewriteProbe(pod.Annotations, pod, req.meshConfig.GetDefaultConfig().GetStatusPort())
    606         }
    607         return nil
    608 }

    622 func (s *Server) handleAppProbeHTTPGet(w http.ResponseWriter, req *http.Request, prober *Prober, path string) {
    623         proberPath := prober.HTTPGet.Path
    624         if !strings.HasPrefix(proberPath, "/") {
    625                 proberPath = "/" + proberPath
    626         }
                   ...
    663         // get the http client must exist because
    664         httpClient := s.appProbeClient[path]
    665 
    666         // Send the request.
    667         response, err := httpClient.Do(appReq)
    668         if err != nil {
    669                 log.Errorf("Request to probe app failed: %v, original URL path = %v\napp URL path = %v", err, path, proberPath)
    670                 w.WriteHeader(http.StatusInternalServerError)
    671                 return
    672         }
	    673         defer func() {
    674                 // Drain and close the body to let the Transport reuse the connection
    675                 _, _ = io.Copy(io.Discard, response.Body)
    676                 _ = response.Body.Close()
    677         }()
    678 
    679         if isRedirect(response.StatusCode) { // Redirect
    680                 // In other cases, we return the original status code. For redirects, it is illegal to
    681                 // not have Location header, so we need to switch to just 200.
    682                 w.WriteHeader(http.StatusOK)
    683                 return
    684         }
    685         // We only write the status code to the response.
    686         w.WriteHeader(response.StatusCode)
    687 }

Request to probe app failed: Get "http://xxx/ok.jsp": read tcp 127.0.0.6:58949->xxx: read: connection reset by peer, original URL path = /app-health/jetty/livez
app URL path = /ok.jsp