I spent few more hours and trying to find the difference between 2 files Working log file

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "a2355547-3f75-46f0-8084-f0741210c892",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/vmware-system-csi/pods",
  "verb": "deletecollection",
  "user": {
    "username": "system:serviceaccount:kube-system:namespace-controller",
    "uid": "8ff8ce11-2bc1-4071-ba6c-6378a0f1cc93",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "10.0.0.176"
  ],
  "userAgent": "kube-controller-manager/v1.28.3 (linux/amd64) kubernetes/a8a1abc/system:serviceaccount:kube-system:namespace-controller",
  "objectRef": {
    "resource": "pods",
    "namespace": "vmware-system-csi",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2024-03-09T08:57:11.838483Z",
  "stageTimestamp": "2024-03-09T08:57:11.839564Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:controller:namespace-controller\" of ClusterRole \"system:controller:namespace-controller\" to ServiceAccount \"namespace-controller/kube-system\""
  }
}

Non working file

{
  "category": "kube-audit",
  "operationName": "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
  "properties": {
    "stream": "stdout",
    "pod": "kube-apiserver-75b4b75759-5v5lx",
    "log": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"0daad20e-d916-4133-b204-6a48d4eda6e6\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/api/v1/serviceaccounts?limit=500\\u0026resourceVersion=0\",\"verb\":\"list\",\"user\":{\"username\":\"system:apiserver\",\"uid\":\"9780069e-d923-43ce-b39b-7d961a904f45\",\"groups\":[\"system:masters\"]},\"sourceIPs\":[\"127.0.0.1\"],\"userAgent\":\"kube-apiserver/v1.27.9 (linux/amd64) kubernetes/d15213f\",\"objectRef\":{\"resource\":\"serviceaccounts\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2024-03-09T06:29:15.156546Z\",\"stageTimestamp\":\"2024-03-09T06:29:15.253005Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"\"}}\n",
    "containerID": "57cb35a25c565fc468381b49dc2eb67dda586dc13f6fcaf20f14088f9ac5ad81"
  },
  "resourceId": "/SUBSCRIPTIONS/XXXXX",
  "serviceBuild": "na",
  "time": "2024-03-09T06:29:15.253110865Z"
}

I noticed something that logs start in non working json file there logs json object iside n json object but I am not sure if that difference is causing the logs to not show up Not working -

"log":"{\"kind\":\"Event\"

Working:

{"kind":"Event","

Log file content difference shouldn't cause dropping of the logs unless you have some filter in place to do so. @nitya-signoz can you please look into this?

we were already discussing it here @Prashant Shahi https://signoz-community.slack.com/archives/C01HWUTP4HH/p1710011370078749