https://signoz.io logo
Join Slack
Powered by
Hi everyone. I'm evaluating SigNoz for observabil...
# support
g
Hi everyone. I'm evaluating SigNoz for observability and APM on a on-premise, no interne, infrastructure in production. Now I want to implement everything on a sandbox VM. After clickhouse install I did clickhouse-client -q "SELECT version()" -> 25.3.2.39 Then
Copy code
clickhouse-client -q "CREATE USER signoz IDENTIFIED BY 'signoz';" 
clickhouse-client -q "GRANT ALL ON signoz_traces.* TO signoz;"
clickhouse-client -q "GRANT ALL ON signoz_logs.* TO signoz;"
clickhouse-client -q "GRANT ALL ON signoz_metrics.* TO signoz;"

clickhouse-client -q "CREATE DATABASE IF NOT EXISTS signoz_traces;"
clickhouse-client -q "CREATE DATABASE IF NOT EXISTS signoz_logs;"
clickhouse-client -q "CREATE DATABASE IF NOT EXISTS signoz_metrics;"

clickhouse-client --user signoz --password signoz
Connection ok FOr SigNoz itself, once installed (following the documentation)
Copy code
sudo bash -c 'cat <<EOF > /opt/signoz/conf/systemd.env
SIGNOZ_INSTRUMENTATION_LOGS_LEVEL=info
INVITE_EMAIL_TEMPLATE=/opt/signoz/templates/invitation_email_template.html
SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
SIGNOZ_WEB_ENABLED=true
SIGNOZ_WEB_DIRECTORY=/opt/signoz/web
SIGNOZ_JWT_SECRET=secret
SIGNOZ_ALERTMANAGER_PROVIDER=signoz
SIGNOZ_TELEMETRYSTORE_PROVIDER=clickhouse
SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=<tcp://localhost:9000?password=signoz>
EOF'
Copy code
sudo getent passwd signoz >/dev/null || sudo useradd --system --home /opt/signoz --no-create-home --user-group --shell /sbin/nologin signoz
sudo chown -R signoz:signoz /var/lib/signoz
sudo chown -R signoz:signoz /opt/signoz
Copy code
sudo bash -c 'cat <<EOF > /etc/systemd/system/signoz.service
[Unit]
Description=SigNoz
Documentation=<https://signoz.io/docs>
After=clickhouse-server.service

[Service]
User=signoz
Group=signoz
Type=simple
KillMode=mixed
Restart=on-failure
WorkingDirectory=/opt/signoz
EnvironmentFile=/opt/signoz/conf/systemd.env
ExecStart=/opt/signoz/bin/signoz --config=/opt/signoz/conf/prometheus.yml --use-logs-new-schema=true --use-trace-new-schema=true

[Install]
WantedBy=multi-user.target
EOF'
I have copy pasted on the dashboard the template for hostmetrics but I got.
Copy code
code: 516, message: default: Authentication failed: password is incorrect, or there is no user with such name. If you use ClickHouse Cloud, the password can be reset at <https://clickhouse.cloud/> on the settings page for the corresponding service. If you have installed ClickHouse and forgot password you can reset it in the configuration file. The password for default user is typically located at /etc/clickhouse-server/users.d/default-password.xml and deleting this file will reset the password. See also /etc/clickhouse-server/users.xml on the server where ClickHouse is installed.
I may made a typo somewhere, but I think i have followed the how-to. ANy idea here ?
v
Hey @Gil, Can you try: SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://localhost:9000?username=signoz&amp;password=signoz
g
Thank you @Vibhu Pandey! username and not user ! Now I got anotther error code: 60, message: Unknown table expression identifier 'signoz_metrics.distributed_time_series_v4_1day' in scope SELECT JSONExtractString(labels, 'host_name') AS host_name FROM signoz_metrics.distributed_time_series_v4_1day WHERE metric_name = 'system_cpu_time' GROUP BY host_name
I see on my clickhouse table with 
SHOW TABLES FROM signoz_metrics;
that I have nothing in it. My otelcol maybe not well configured
v
did you run the schema migrations?
g
I don't know what is the schema migration ? I just followed the tutorial, adapting on my use case, copy/paste the hostmetrics template
@Vibhu Pandey I just reproof the how to and yes I jump the migration part. What I understand is that bin create the needed tables So I tried
Copy code
./schema-migrator sync --dsn="<tcp://localhost:9000?username=signoz&password=signoz>"                                                                                  
{"L":"info","timestamp":"2025-04-02T15:12:05.966+0200","C":"signozschemamigrator/main.go:91","M":"Running migrations in sync mode","dsn":"<tcp://localhost:9000?username=signoz&password=signoz>","replication":f
alse,"cluster-name":"cluster"}                                                                                                                                                                                 
{"L":"info","timestamp":"2025-04-02T15:12:05.966+0200","C":"signozschemamigrator/main.go:104","M":"Up migrations","versions":[]}                                                                               
{"L":"info","timestamp":"2025-04-02T15:12:05.966+0200","C":"signozschemamigrator/main.go:117","M":"Down migrations","versions":[]}                                                                             
{"L":"info","timestamp":"2025-04-02T15:12:05.966+0200","C":"signozschemamigrator/main.go:127","M":"Parsed DSN","optsError":"json: unsupported type: func(context.Context, string) (net.Conn, error)"}          
{"L":"info","timestamp":"2025-04-02T15:12:05.967+0200","C":"signozschemamigrator/main.go:133","M":"Opened connection"}                                                                                         
Error: failed to bootstrap migrations: failed to create dbs                                                                                                                                                    
failed to create dbs                                                                                                                                                                                           
code: 701, message: Requested cluster 'cluster' not found                                                                                                                                                      
Usage:                                                                                                                                                                                                         
  signoz-schema-migrator sync [flags]                                                                                                                                                                          
                                                                                                                                                                                                               
Flags:                                                                                                                                                                                                         
      --down string   Down migrations to run, comma separated. Must provide down migrations explicitly to run                                                                                                  
  -h, --help          help for sync                                                                                                                                                                            
      --up string     Up migrations to run, comma separated. Leave empty to run all up migrations                                                                                                              
                                                                                                                                                                                                               
Global Flags:                                                                                                                                                                                                  
      --cluster-name string   Cluster name to use while running migrations (default "cluster")                                                                                                                 
      --dev                   Development mode                                                                                                                                                                 
      --dsn string            Clickhouse DSN                                                                                                                                                                   
      --replication           Enable replication
Cluster not found
v
Nice, were you able to follow the clickhouse instructions where we add a custom xml file to create a cluster?
g
Hi @Vibhu Pandey, I'm into understanding the config file. Not sure what's wrong with the vanilla conf
v
Step 9 of install zookeeper: https://signoz.io/docs/install/linux/#install-zookeeper A 
cluster.xml
file creates the cluster to talk to.
g
I didn't install zookeeper since I'm going one node. Don't think I will need distributed DB in a short term
v
Ahhh we only have distributed support right now. That's why the errors.
g
Oh I cannot install with mono node ?
v
yesss
g
That's change everything. I'm not sure about my client final needs. Now it's low but it's gonna evolve for sure. So what's the minimal recomanded setup ? clickhouse x3 + zookeeper to manage the cluster, signoz back and front service right ?
v
Clickhouse + Zookeeper SigNoz Otel Collector SigNoz binary
g
You right I forgot the collector. SO how many VM ? I cannot install everything on one machine ?
v
Depending on your scale, I would suggest starting out with 1 machine and assessing
g
I still can use one node clickhouse but handled by zookeeper. That's what you mean ?
8CPU / 32 RAM seems enough for a start ? Also whats is the purpose of "Install standalone ClickHouse Keeper"
v
Yes you can still use 1 clickhouse node, 1 zookeeper.
You don't need Clickhouse keeper when you have zookeeper running. I'm assuming you are reading "Install standalone ClickHouse Keeper" from the clickhouse docs?
g
I get it, just what is the purpose ok keeper ? I want to know.
Yes I'm following this : https://signoz.io/docs/install/linux I cannot use docker and I'm on redhat 8.10
v
Perfect. Keeper is a lightweight replacement to zookeeper. That's all.
You can skip installing clickhouse keeper as we are yet to benchmark it and test it with SigNoz.
g
Ok it's like Kafka that have ditched ZK also for their own tech in the v4
v
Yup similar
g
I just get an unused VM. I will reinstall everything from scratch. ZK included.
Thank you very much for our support. Are you working for the company behind SigNoz ?
v
Yes!
g
I have followd the how to but zookeeper is down
Copy code
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: <https://access.redhat.com/support>
-- 
-- The unit zookeeper.service has entered the 'failed' state with result 'exit-code'.
Apr 04 12:03:46 FRPAR3DXSBOX systemd[1]: Failed to start Zookeeper.
-- Subject: Unit zookeeper.service has failed
-- Defined-By: systemd
-- Support: <https://access.redhat.com/support>
-- 
-- Unit zookeeper.service has failed.
-- 
-- The result is failed.
Apr 04 12:03:46 FRPAR3DXSBOX systemd[1]: zookeeper.service: Service RestartSec=100ms expired, scheduling restart.
Apr 04 12:03:46 FRPAR3DXSBOX systemd[1]: zookeeper.service: Scheduled restart job, restart counter is at 5.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: <https://access.redhat.com/support>
-- 
-- Automatic restarting of the unit zookeeper.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Apr 04 12:03:46 FRPAR3DXSBOX systemd[1]: Stopped Zookeeper.
-- Subject: Unit zookeeper.service has finished shutting down
-- Defined-By: systemd
-- Support: <https://access.redhat.com/support>
-- 
-- Unit zookeeper.service has finished shutting down.
Apr 04 12:03:46 FRPAR3DXSBOX systemd[1]: zookeeper.service: Start request repeated too quickly.
Apr 04 12:03:46 FRPAR3DXSBOX systemd[1]: zookeeper.service: Failed with result 'exit-code'.
v
Was java installeD?
g
Which version of java is needed ?
v
Copy code
ZooKeeper runs in Java, release 1.8 or greater (JDK 8 LTS, JDK 11 LTS, JDK 12
Copy code
Java 9 and 10 are not supported
g
Ok so 17 is LTS and we use it
Hey @Vibhu Pandey I'm into the migration step. This is to create the table inside CH, right ?
ALso what are the difference between the sync and async migration ?
And running the migration tool I have those errors
Copy code
./signoz-schema-migrator_linux_amd64/bin/signoz-schema-migrator: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./signoz-schema-migrator_linux_amd64/bin/signoz-schema-migrator)
./signoz-schema-migrator_linux_amd64/bin/signoz-schema-migrator: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./signoz-schema-migrator_linux_amd64/bin/signoz-schema-migrator)
I think I need to build the binaries myself
ldd --version -> 2.28
RedHat and co are compiled with 2.28 so the version you offered are not usable on those OS
I have created a "signoz" user to clickhouse to not let the default user play a role in this. I have to grant a lot of different stuff. Right now I have done
Copy code
clickhouse-client -q "GRANT CLUSTER ON *.* TO signoz;"
clickhouse-client -q "GRANT CREATE DATABASE ON signoz_metadata.* TO signoz;"
clickhouse-client -q "GRANT CREATE DATABASE ON signoz_analytics.* TO signoz;"
clickhouse-client -q "GRANT SELECT(host_address, port, cluster) ON system.clusters TO signoz;"
clickhouse-client -q "GRANT SELECT(entry, cluster, query, host, port, status, exception_code) ON system.distributed_ddl_queue TO signoz;"
And I'm not finish. Is there a simple "ADMIN" role that I can give to my signoz user ?
HI @Vibhu Pandey How are you today ?
14 Views
Open in Slack
PreviousNext
Powered by