Hi Team as we tried to upgrade signoz with chart version 0 4 SigNoz Community #support

Hi Team, as we tried to upgrade signoz with chart ...

Yi Luo

07/19/2024, 2:54 PM

Hi Team, as we tried to upgrade signoz with chart version 0.44.1 to 0.45.3 we had migration issue for click house, any help would be much appreciated, Thank you

{"level":"fatal","timestamp":"2024-07-19T14:30:17.010Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 10. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go:267"}

✅ 1

Prashant Shahi

07/19/2024, 3:06 PM

@nitya-signoz can you help here?

nitya-signoz

07/19/2024, 3:17 PM

How many clickhouse shards are you running ?

Yi Luo

07/19/2024, 3:28 PM

We use the default number of clickhouse shards, so I assume it is one shard

nitya-signoz

07/19/2024, 3:31 PM

Got it, so the 10th migration is a resource heavy migration so sometimes the migration timesouts and fails even though the query ran. Can you share the result of

show create table signoz_logs.logs

Yi Luo

07/19/2024, 3:34 PM

Thank you for the help. Here is the result:

Copy code

Query id: ac17d601-98df-47b0-a488-d30760bdaf0a

┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE signoz_logs.logs
(
    `timestamp` UInt64 CODEC(DoubleDelta, LZ4),
    `observed_timestamp` UInt64 CODEC(DoubleDelta, LZ4),
    `id` String CODEC(ZSTD(1)),
    `trace_id` String CODEC(ZSTD(1)),
    `span_id` String CODEC(ZSTD(1)),
    `trace_flags` UInt32,
    `severity_text` LowCardinality(String) CODEC(ZSTD(1)),
    `severity_number` UInt8,
    `body` String CODEC(ZSTD(2)),
    `resources_string_key` Array(String) CODEC(ZSTD(1)),
    `resources_string_value` Array(String) CODEC(ZSTD(1)),
    `attributes_string_key` Array(String) CODEC(ZSTD(1)),
    `attributes_string_value` Array(String) CODEC(ZSTD(1)),
    `attributes_int64_key` Array(String) CODEC(ZSTD(1)),
    `attributes_int64_value` Array(Int64) CODEC(ZSTD(1)),
    `attributes_float64_key` Array(String) CODEC(ZSTD(1)),
    `attributes_float64_value` Array(Float64) CODEC(ZSTD(1)),
    `log` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'log')] CODEC(LZ4),
    `attribute_string_martello_env` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'martello_env')] CODEC(LZ4),
    `resource_string_host_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'host_name')] CODEC(LZ4),
    `resource_string_os_type` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'os_type')] CODEC(LZ4),
    `attribute_string_exception_message` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'exception_message')] CODEC(LZ4),
    `attribute_string_exception_stacktrace` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'exception_stacktrace')] CODEC(LZ4),
    `attribute_string_dotnet_ilogger_category` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'dotnet_ilogger_category')] CODEC(LZ4),
    `resource_string_service_version` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'service_version')] CODEC(LZ4),
    `resource_string_service_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'service_name')] CODEC(LZ4),
    `resource_string_k8s_cluster_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_cluster_name')] CODEC(LZ4),
    `attribute_string_martello_region` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'martello_region')] CODEC(LZ4),
    `resource_string_k8s_namespace_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_namespace_name')] CODEC(LZ4),
    `resource_string_k8s_node_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_node_name')] CODEC(LZ4),
    `resource_string_vdx_tenant_id` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'vdx_tenant_id')] CODEC(LZ4),
    `resource_string_vdx_tenant_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'vdx_tenant_name')] CODEC(LZ4),
    `resource_string_k8s_container_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_container_name')] CODEC(LZ4),
    `attribute_string_exception_message_exists` Bool MATERIALIZED if(indexOf(attributes_string_key, 'exception_message') != 0, true, false) CODEC(ZSTD(1)),
    `attribute_string_martello_env_exists` Bool MATERIALIZED if(indexOf(attributes_string_key, 'martello_env') != 0, true, false) CODEC(ZSTD(1)),
    `attribute_string_exception_stacktrace_exists` Bool MATERIALIZED if(indexOf(attributes_string_key, 'exception_stacktrace') != 0, true, false) CODEC(ZSTD(1)),
    `attribute_string_dotnet_ilogger_category_exists` Bool MATERIALIZED if(indexOf(attributes_string_key, 'dotnet_ilogger_category') != 0, true, false) CODEC(ZSTD(1)),
    `attribute_string_martello_region_exists` Bool MATERIALIZED if(indexOf(attributes_string_key, 'martello_region') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s_container_name_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'k8s_container_name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_host_name_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'host_name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s_cluster_name_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'k8s_cluster_name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s_node_name_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'k8s_node_name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_vdx_tenant_id_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'vdx_tenant_id') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_os_type_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'os_type') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s_namespace_name_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'k8s_namespace_name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_vdx_tenant_name_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'vdx_tenant_name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_service_name_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'service_name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_service_version_exists` Bool MATERIALIZED if(indexOf(resources_string_key, 'service_version') != 0, true, false) CODEC(ZSTD(1)),
    `attributes_bool_key` Array(String) CODEC(ZSTD(1)),
    `attributes_bool_value` Array(Bool) CODEC(ZSTD(1)),
    `attribute_string_exception$$message` String DEFAULT attributes_string_value[indexOf(attributes_string_key, 'exception.message')] CODEC(ZSTD(1)),
    `attribute_string_exception$$message_exists` Bool DEFAULT if(indexOf(attributes_string_key, 'exception.message') != 0, true, false) CODEC(ZSTD(1)),
    `attribute_string_exception$$stacktrace` String DEFAULT attributes_string_value[indexOf(attributes_string_key, 'exception.stacktrace')] CODEC(ZSTD(1)),
    `attribute_string_exception$$stacktrace_exists` Bool DEFAULT if(indexOf(attributes_string_key, 'exception.stacktrace') != 0, true, false) CODEC(ZSTD(1)),
    `attribute_string_dotnet$$ilogger$$category` String DEFAULT attributes_string_value[indexOf(attributes_string_key, 'dotnet.ilogger.category')] CODEC(ZSTD(1)),
    `attribute_string_dotnet$$ilogger$$category_exists` Bool DEFAULT if(indexOf(attributes_string_key, 'dotnet.ilogger.category') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s$$namespace$$name` String DEFAULT resources_string_value[indexOf(resources_string_key, 'k8s.namespace.name')] CODEC(ZSTD(1)),
    `resource_string_k8s$$namespace$$name_exists` Bool DEFAULT if(indexOf(resources_string_key, 'k8s.namespace.name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s$$cluster$$name` String DEFAULT resources_string_value[indexOf(resources_string_key, 'k8s.cluster.name')] CODEC(ZSTD(1)),
    `resource_string_k8s$$cluster$$name_exists` Bool DEFAULT if(indexOf(resources_string_key, 'k8s.cluster.name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_service$$version` String DEFAULT resources_string_value[indexOf(resources_string_key, 'service.version')] CODEC(ZSTD(1)),
    `resource_string_service$$version_exists` Bool DEFAULT if(indexOf(resources_string_key, 'service.version') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_vdx$$tenant$$name` String DEFAULT resources_string_value[indexOf(resources_string_key, 'vdx.tenant.name')] CODEC(ZSTD(1)),
    `resource_string_vdx$$tenant$$name_exists` Bool DEFAULT if(indexOf(resources_string_key, 'vdx.tenant.name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_os$$type` String DEFAULT resources_string_value[indexOf(resources_string_key, 'os.type')] CODEC(ZSTD(1)),
    `resource_string_os$$type_exists` Bool DEFAULT if(indexOf(resources_string_key, 'os.type') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_host$$name` String DEFAULT resources_string_value[indexOf(resources_string_key, 'host.name')] CODEC(ZSTD(1)),
    `resource_string_host$$name_exists` Bool DEFAULT if(indexOf(resources_string_key, 'host.name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_vdx$$tenant$$id` String DEFAULT resources_string_value[indexOf(resources_string_key, 'vdx.tenant.id')] CODEC(ZSTD(1)),
    `resource_string_vdx$$tenant$$id_exists` Bool DEFAULT if(indexOf(resources_string_key, 'vdx.tenant.id') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s$$node$$name` String DEFAULT resources_string_value[indexOf(resources_string_key, 'k8s.node.name')] CODEC(ZSTD(1)),
    `resource_string_k8s$$node$$name_exists` Bool DEFAULT if(indexOf(resources_string_key, 'k8s.node.name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_service$$name` String DEFAULT resources_string_value[indexOf(resources_string_key, 'service.name')] CODEC(ZSTD(1)),
    `resource_string_service$$name_exists` Bool DEFAULT if(indexOf(resources_string_key, 'service.name') != 0, true, false) CODEC(ZSTD(1)),
    `resource_string_k8s$$container$$name` String DEFAULT resources_string_value[indexOf(resources_string_key, 'k8s.container.name')] CODEC(ZSTD(1)),
    `resource_string_k8s$$container$$name_exists` Bool DEFAULT if(indexOf(resources_string_key, 'k8s.container.name') != 0, true, false) CODEC(ZSTD(1)),
    INDEX id_minmax id TYPE minmax GRANULARITY 1,
    INDEX log_idx log TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX severity_number_idx severity_number TYPE set(25) GRANULARITY 4,
    INDEX severity_text_idx severity_text TYPE set(25) GRANULARITY 4,
    INDEX trace_flags_idx trace_flags TYPE bloom_filter GRANULARITY 4,
    INDEX attribute_string_exception_message_idx attribute_string_exception_message TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX attribute_string_martello_env_idx attribute_string_martello_env TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX attribute_string_exception_stacktrace_idx attribute_string_exception_stacktrace TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX attribute_string_dotnet_ilogger_category_idx attribute_string_dotnet_ilogger_category TYPE bloom_fil⋯│

nitya-signoz

07/19/2024, 3:38 PM

ahh this doesn’t contain the entire data can you run

show create table signoz_logs.logs format Vertical

Yi Luo

07/19/2024, 3:40 PM

Here it is, result too long, put into a txt file

result.txt

nitya-signoz

07/19/2024, 3:47 PM

Hmm, so seems like a part of this migrations ran https://github.com/SigNoz/signoz-otel-collector/blob/main/migrationmanager/migrators/logs/migrations/000010_body_ngram.up.sql Next steps:- • Check if there are any mutations running

select * from system.mutations where is_done=0

• If there above query result is empty run

ALTER TABLE signoz_logs.logs ON CLUSTER cluster ADD INDEX IF NOT EXISTS body_idx lower(body) TYPE ngrambf_v1(4, 60000, 5, 0) GRANULARITY 1;

• Once the above is complete run

alter table signoz_logs.schema_migrations update dirty=0 where version=10

✅ 1

Yi Luo

07/19/2024, 3:53 PM

I have done the above steps, I guess the next step would be try upgrade signoz chart version from 0.44.1 to 0.45.3 again?

Yi Luo

07/19/2024, 3:56 PM

Or do I update one minor version at a time?

nitya-signoz

07/19/2024, 4:01 PM

that is not required you can directly try upgrading

✅ 1

Yi Luo

07/19/2024, 4:06 PM

Got a new error, when I tried to re-applied the 0.44.1

Copy code

"msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: no migration found for version 10: read down for version 10 .: file does not exist","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go:267"}

Yi Luo

07/19/2024, 4:07 PM

Should I directly do the upgrade instead of reapplying the current version?

nitya-signoz

07/19/2024, 4:11 PM

yes correct

Yi Luo

07/19/2024, 4:29 PM

Tried to upgrade to 0.45.3 again got this now:

Copy code

{
  "level": "fatal",
  "timestamp": "2024-07-19T16:27:12.020Z",
  "caller": "signozschemamigrator/migrate.go:128",
  "msg": "Failed to run migrations",
  "component": "migrate cli",
  "error": "clickhouse migrate failed to run, error: Dirty database version 12. Fix and force version.",
  "stacktrace": "main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go:267"
}

nitya-signoz

07/19/2024, 4:32 PM

Can you show me your schema again ? https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1721403501455299?thread_ts=1721400860.382799&cid=C01HWQ1R0BC

nitya-signoz

07/19/2024, 4:36 PM

So basically you will have to run the migration 12 and 13 manually • https://github.com/SigNoz/signoz-otel-collector/blob/main/migrationmanager/migrators/logs/migrations/000012_rename_instrumentation_scope.up.sql • https://github.com/SigNoz/signoz-otel-collector/blob/main/migrationmanager/migrators/logs/migrations/000013_rename_instrumentation_scope.up.sql and update the schema migrations table by adding the versions manually. you can run the above by replacing

{{.SIGNOZ_CLUSTER}}

with

cluster

.Let me know once done ------ once the above is done, it means that you have done the migrations manually, now you will have to update the schema migrations table so that those migrations don’t run again. Here are the steps for that •

truncate table signoz_logs.schema_migrations

•

insert into signoz_logs.schema_migrations values (1 ,1 ,1720103647412982569), (1 ,0 ,1720103647810031709), (2 ,1 ,1720103647811955242), (2 ,0 ,1720103647869983948), (3 ,1 ,1720103647875299543), (3 ,0 ,1720103648053893053), (4 ,1 ,1720103648055714316), (4 ,0 ,1720103648113219798), (5 ,1 ,1720103648115626118), (5 ,0 ,1720103648678466995), (6 ,1 ,1720103648680332053), (6 ,0 ,1720103648795967894), (7 ,1 ,1720103648797859904), (7 ,0 ,1720103648967002426), (8 ,1 ,1720103648969563518), (8 ,0 ,1720103649251171410), (9 ,1 ,1720103649252821091), (9 ,0 ,1720103649365897523), (10 ,1 ,1720103649367680188), (10 ,0 ,1720103660451466954), (11 ,1 ,1720103660454458733), (11 ,0 ,1720103661073573745), (12 ,1 ,1720115099220469277), (12 ,0 ,1720115146203106526), (13 ,1 ,1720115146205945060), (13 ,0 ,1720115146319582687)

✅ 1

🙌 1

Yi Luo

07/19/2024, 5:46 PM

@nitya-signoz Great! By following your steps, the upgrade successful. Thank you so much

🎉 1

33 Views

Open in Slack

Previous Next