Troubleshooting

This page contains troubleshooting tips to help find out the reason of issues.

Slurmrestd

Test Slurm slurmrestd API is properly responding on Unix socket with these commands:

JWT auth on Unix socket
JWT auth on TCP/IP socket
local auth

When using slurmrestd JWT authentication on listening on Unix socket:

$ export $(scontrol token)
$ curl --silent --header X-SLURM-USER-TOKEN:$SLURM_JWT --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v0.0.41/diag | \
  jq '.statistics | with_entries(select(.key | startswith("jobs")))'
{
  "jobs_submitted": 385,
  "jobs_started": 407,
  "jobs_completed": 411,
  "jobs_canceled": 0,
  "jobs_failed": 0,
  "jobs_pending": 0,
  "jobs_running": 0
}

When using slurmrestd JWT authentication on listening on TCP/IP socket:

$ export $(scontrol token)
$ curl --silent --header X-SLURM-USER-TOKEN:$SLURM_JWT http://localhost:6820/slurm/v0.0.41/diag | \
  jq '.statistics | with_entries(select(.key | startswith("jobs")))'
{
  "jobs_submitted": 385,
  "jobs_started": 407,
  "jobs_completed": 411,
  "jobs_canceled": 0,
  "jobs_failed": 0,
  "jobs_pending": 0,
  "jobs_running": 0
}

When using slurmrestd local authentication (deprecated):

$ curl --silent --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v0.0.41/diag | \
  jq '.statistics | with_entries(select(.key | startswith("jobs")))'
{
  "jobs_submitted": 385,
  "jobs_started": 407,
  "jobs_completed": 411,
  "jobs_canceled": 0,
  "jobs_failed": 0,
  "jobs_pending": 0,
  "jobs_running": 0
}

This command should print JSON output with current jobs statistics on the cluster.

Test Slurm accounting on in REST API with this command:

JWT auth on Unix socket
JWT auth on TCP/IP socket
local auth

When using slurmrestd JWT authentication on listening on Unix socket:

$ curl --silent --header X-SLURM-USER-TOKEN:$SLURM_JWT --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurmdb/v0.0.41/config | \
  jq .clusters[].nodes
"cn[1-4]"

When using slurmrestd JWT authentication on listening on TCP/IP socket:

$ curl --silent --header X-SLURM-USER-TOKEN:$SLURM_JWT http://localhost:6820/slurmdb/v0.0.41/config | \
  jq .clusters[].nodes
"cn[1-4]"

When using slurmrestd local authentication (deprecated):

$ curl --silent --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurmdb/v0.0.41/config | \
  jq .clusters[].nodes
"cn[1-4]"

This command should print the set of compute nodes in the cluster.

Logs of slurmrestd are available with this command:

# journalctl --unit slurmrestd.service

Informational and debug messages can be filtered out to see only the errors with this command:

# journalctl --priority=notice --unit slurmrestd.service

Native Services

This section provides instructions to troubleshoot Slurm-web when running with native services (ie. slurm-web-gateway.service and slurm-web-agent.service).

Test Slurm-web gateway API is available with this command:

$ curl http://localhost:5012/api/version
Slurm-web gateway v7.0.0

Test Slurm-web agent API is available with this command:

$ curl http://localhost:5013/version
Slurm-web agent v7.0.0

Logs of native services are available with these commands:

# journalctl --unit slurm-web-agent.service
# journalctl --unit slurm-web-gateway.service

WSGI Services

This section provides instructions to troubleshoot Slurm-web when running as WSGI applications on production HTTP servers.

Test Slurm-web gateway API is available with this command:

$ curl http://localhost/api/version
Slurm-web gateway v7.0.0

Test Slurm-web agent API is available with this command:

$ curl http://localhost/agent/version
Slurm-web agent v7.0.0

Logs of uWSGI services are available with these commands:

# journalctl --unit slurm-web-agent-uwsgi.service
# journalctl --unit slurm-web-gateway-uwsgi.service

Check for possible errors logs of HTTP servers:

Nginx

In file /var/log/nginx/error.log

Apache2

On Debian/Ubuntu: In file /var/log/apache2/error.log
On RHEL (and compatible) and Fedora: /var/log/httpd/error_log

Caddy

Run this command:

# journalctl --unit caddy.service

Containers

This section provides instructions to troubleshoot Slurm-web when running with OCI containers or the reference Compose stack.

The following commands assume the default ports published by the container quickstarts: TCP/5011 for the gateway and TCP/5012 for the agent.

Test Slurm-web gateway API is available with this command:

$ curl http://localhost:5011/api/version
Slurm-web gateway v7.0.0

Test Slurm-web agent API is available with this command:

$ curl http://localhost:5012/version
Slurm-web agent v7.0.0

Slurm-web container images run Gunicorn with access and error logs written to standard output and standard error. Container runtime logs therefore include HTTP access logs and application errors.

Docker
Podman
Docker Compose
Podman Compose

$ docker logs slurm-web-gateway
$ docker logs slurm-web-agent

$ podman logs slurm-web-gateway
$ podman logs slurm-web-agent

$ docker compose logs gateway agent

$ podman compose logs gateway agent

Use the -f option to follow logs in real time, for example:

$ docker logs -f slurm-web-gateway
$ docker compose logs -f gateway agent

Authentication

When users cannot sign in through the gateway, see the dedicated troubleshooting guides for each authentication method:

LDAP
OIDC

Authorization Policy

To help understand roles and permissions granted by authorization policy on clusters, users can go in menu:Settings[Account] to view their permissions on clusters. For example:

In this example, the user cingram is member of users and biology groups in LDAP directory.

On cluster emulator, he is assigned roles special and user with permissions on jobs-view, qos-view and stats-view actions.

On cluster tiny, he is assigned roles admin and users with permissions on accounts-view, jobs-view, nodes-view, partitions-view, qos-view, reservations-view and stats-view actions.