Investigating when Graph Studio is Unresponsive

If the Graph Studio server seems unresponsive or crashes, it is important to gather diagnostic information while the server is in the unresponsive state to aid in the investigation of the issue. This topic describes some of the symptoms you may encounter and includes recommendations on retrieving information to help diagnose the issue.

Symptom

The Hi-Res Analytics and Graph Studio applications are inaccessible. When trying to reach the applications, you experience one of the following symptoms:

  • A "404 not found" error.
  • A "server connection was lost" message, and clicking refresh does not reconnect to the server.
  • Pages remain blank and never load.

Steps for Investigating the Cause

The suggestions below provide options for investigating the issue and collecting diagnostic information to review and, if necessary, send to Cambridge Semantics Support for further investigation.

Make sure that Anzo is running

First, it is helpful to make sure that Graph Studio is still running. From the command line on the Graph Studio server, you can run the following command to check whether the anzo process is running: ps aux | grep anzo. If the process is not found, Graph Studio is not running and can be restarted with sudo systemctl start anzo.

Check basic resource utilization

Next, check basic resource utilization to determine whether the server is close to the limit of memory or disk resources. From the command line, you can run the following commands:

  • top: To display CPU usage and threads.
  • free g: To display memory usage.
  • df -h: To display disk usage.

If disks are close to full or memory usage is close to the limit, you may need to increase the resources on the server or delete files to reclaim disk space.

There are ways that you can limit the disk space that is used to store logs and unstructured status journals if you run unstructured pipelines. See topics such as Limiting the Age/Size of Audit Logs, Limiting the Size/Number of anzo_full Logs, and Limiting the Number of Unstructured Status Journals in Advanced Semantic Service Configuration.

Adding more memory to your machine does not automatically allocate more memory to Graph Studio. To increase memory for Graph Studio if you have added memory to the server, modify the -XmxNNNm value in the AnzoServer.vmoptions file in the <install_path>/Server directory. It is important that you do not over-allocate memory. Make sure that the value you specify is no more than 70 to 80 percent of the total memory on the server. After editing the file, restart Graph Studio to configure the change.

View the Anzo log files

Review the Graph Studio log files for errors. The anzo_full.log contains the most comprehensive information, but the _error logs are more focused. For instructions on viewing the logs, see Viewing Log Files.

View the JVM stack

The sysadmin user can view the stack to see the current state of the JVM and look for deadlocks or blocked threads. For instructions, see Viewing the Current Stack in a Browser.

Investigate with the OSGi console

The sysadmin user can also SSH into OSGi console. To do so, follow the steps below:

If the OSGi console is inaccessible, retrieve a thread dump instead. See Generate a thread dump below.

  1. On the Graph Studio server, make sure that you are logged in as the user that is running Graph Studio.
  2. Run the following command to log in to the console:
    ssh -p 8022 sysadmin@127.0.0.1
  3. When prompted, enter the sysadmin password.
  4. At the osgi> prompt, you can type help and press Enter to view the available commands. The list below describes the commands that are helpful for troubleshooting:
    • queries: Lists any active local volume queries being run. This command only reports on active requests against a journal, not Graph Lakehouse. If there are long-running local volume queries, you can run cancelQuery to cancel them.
    • sysinfo: Outputs information about total memory, free memory, max memory, and CPU usage.
    • gc: Means "garbage collect." When run, Graph Studio frees memory that is not being used.
    • ss: Displays a list of all installed bundles and their status. This command can be useful for locating and stopping a bundle that is stuck in the initialization phase and causing problems. To find a particular bundle, you can run ss | grep <search_string_in_the_bundle_name>. For example:
      ss | grep asdl

      Returns the bundle ID, the bundle status, and the bundle name. For example:

      24   ACTIVE   com.cambridgesemantics.anzo.asdl.services_5.4.5.r202311151600

      To stop a bundle, run ss <ID>. For example, ss 24.

    • deadlock /local_file_path/file_name.txt: Outputs to the specified file information about deadlocked threads.
    • blocked /local_file_path/file_name.txt: Outputs to the specified file information about threads that are blocking other threads.
    • topStack /local_file_path/file_name.txt: Outputs to the specified file information about the top thread stacks (according to CPU usage).
    • stack /local_file_path/file_name.txt: Outputs to the specified file information about all thread stacks.
    • heap /local_file_path/file_name.hprof: Outputs a heap dump to the specified file. Though the difference between heap dumps and stack dumps is nuanced, in general heap dumps are useful for diagnosing memory issues if you suspect the issue is memory-related, and stack or thread dumps are useful for diagnosing CPU issues if you suspect the issue is CPU-related.

      Heap dumps are very large. At a minimum, the generated file is several GB in size. Make sure that you save the file to a disk with plenty of available space. In addition, heap dumps may contain sensitive data that you might not want to share outside of your organization. Stack and thread dumps do not contain any sensitive data.

Generate a thread dump

If the OSGi console is unresponsive, you can retrieve a thread dump with a script that is included in the installation. To generate the thread dump files, run the Anzo-detailed-thread-dump.sh script in the <install_path>/Server/scripts directory. For example:

cd /opt/Anzo/Server/scripts
./Anzo-detailed-thread-dump.sh

The script output lists the names of the files that are generated in the scripts directory. For example:

Detailed thread dump information written to the following files:
./anzo-thread-detail-jcmd-dump-2023-12-13-20-39-17.txt
./anzo-thread-detail-native-dump-2023-12-13-20-39-17.txt