Code42 server stalls due to too many open files

Last updated
Save as PDF
Share
1. Share
2. Tweet

Who is this article for?

CrashPlan for Small Business

Incydr, no.

CrashPlan for Enterprise, yes.

Code42 for Enterprise, yes.

CrashPlan for Small Business, no.

This article applies to on-premises authority servers.

Overview

In certain circumstances, heavy user activity may cause Linux Code42 servers to slow or pause backup activity and log the error message "too many open files." This occurs due to open file limits imposed by the Linux operating system. This article explains how to correct this issue by increasing the open file limits for Linux.

Windows is not affected by this issue due to the way it handles open files in memory.

In rare cases, individual Linux user devices with very large backup file selections may also exceed the open files limit. For information about increasing the open files limit on user devices, see Backups Stall Due To Too Many Open Files.

Affects

Code42 servers for Linux

Under the hood

Linux imposes a limit on the number of files a process can have open at any one time. More accurately, the operating system imposes a limit on the number of file descriptors a process can have open at any one time, but for the purposes of this article, the difference isn't significant.

The Code42 server may reach this limit if there is too much backup activity happening at one time. It typically manifests itself by preventing users from backing up to the destination. Archive maintenance jobs may also stall.

Diagnosing

If the Code42 server reaches the operating system's open file limit, logs from the destination include an error in com_backup42_app.log similar to the one below:

Caused by: java.io.FileNotFoundException: /2tb/backups/358017395638843928/cpbf0000000000000035259/cpbdf (Too many open files)

It's possible for this issue to manifest itself in different error messages, but the messages always contain the string "Too many open files." See Sending Logs To Enterprise Support for more information about working with log files.

Recommended solution

The sections below cover how to check and change the open files limit.

Step 1: Check the Code42 server open files limit

To check the open files limit in /proc/[PID]/limits, use the process ID of the Code42 server service.

Use ps to find the process ID:
```
ps aux | grep proserver
```
Use cat to view the limit for the process ID:
```
sudo cat /proc/[PID]/limits
```

In the following example, the Code42 server service has a PID of 4758 and an open files limit of 1024 (shown in bold text).

code42@ubuntu:~$ ps aux | grep proserver
root      4758 67.0 18.2 1554672 187280 pts/2  Sl   09:29   0:03 /opt/proserver/jre/bin/java -Dapp=CPServer -server -Dnetworkaddress.cache.ttl=300 -Djava.net.preferIPv4Stack=true -Ddrools.compiler=JANINO -Dfile.encoding=UTF-8 -Dc42.native.md5.enabled=false -XX:+DisableExplicitGC -XX:+UseAdaptiveGCBoundary -XX:PermSize=256m -XX:MaxPermSize=256m -Xss256k -Xms256m -Xmx1024m -jar /opt/proserver/lib/com.backup42.app.jar -prop conf/conf_proe.properties -config conf/conf_proe.groovy
erik      4771  0.0  0.0   7636   920 pts/2    S+   09:29   0:00 grep --color=auto proserver

code42@ubuntu:~$ sudo cat /proc/4758/limits
Limit                     Soft Limit           Hard Limit           Units    
Max cpu time              unlimited            unlimited            seconds  
Max file size             unlimited            unlimited            bytes    
Max data size             unlimited            unlimited            bytes    
Max stack size            8388608              unlimited            bytes    
Max core file size        0                    unlimited            bytes    
Max resident set          unlimited            unlimited            bytes    
Max processes             unlimited            unlimited            processes
Max open files            1024                 1024                 files    
Max locked memory         65536                65536                bytes    
Max address space         unlimited            unlimited            bytes    
Max file locks            unlimited            unlimited            locks    
Max pending signals       16382                16382                signals  
Max msgqueue size         819200               819200               bytes    
Max nice priority         20                   20                  
Max realtime priority     0                    0                   
Max realtime timeout      unlimited            unlimited            us

Step 2: Increase the Code42 server open files limit

Stop the Code42 server service by running the following command:
```
sudo /opt/proserver/bin/proserver stop
```
Open the /opt/proserver/.proserverrc file in a plain text editor.
If this file does not already exist, create it.

Add the following lines to the file:

# Increase open files limit
ulimit -n 409600

Save the file.
Run the following command to start the Code42 server service.
```
sudo /opt/proserver/bin/proserver start
```