Backups stall due to too many open files
Who is this article for?
Incydr, no.
CrashPlan for Enterprise, no.
Code42 for Enterprise, yes.
CrashPlan for Small Business, no.
Overview
In rare circumstances, very large backup file selections may cause the CrashPlan app for Linux and OS X to stop all backup activity and add the error message "too many open files" to your CrashPlan logs. This occurs due to open file limits imposed by the Linux and OS X operating systems. This article explains how to correct this issue by increasing the open file limits for Linux and OS X.
Windows is not affected by this issue due to the way it handles open files in memory.
Affects
The CrashPlan app for Linux and OS X
Under the hood
Linux and OS X impose a limit on the number of files a process can have open at any one time. More accurately, the operating system imposes a limit on the number of file descriptors a process can have open at any one time; for the purposes of this article, the difference isn't significant.
In rare circumstances, the CrashPlan app may reach this limit if your backup file selection contains a very large number of files or if another computer with a very large backup file selection is backing up to your computer. The issue most often occurs during maintenance or the file verification scan. If the CrashPlan app reaches this limit, backup activity may stop.
This issue is not related to the limits on inotify watches that occasionally arise on Linux.
Diagnosing
If the CrashPlan app reaches the open file limit, the CrashPlan logs from your computer will include an error in service.log similar to the one below:
Caused by: java.io.FileNotFoundException: /2tb/backups/358017395638843928/cpbf0000000000000035259/cpbdf (Too many open files)
It's possible for this issue to manifest itself in different error messages, but the messages always contain the string "Too many open files." See Reading CrashPlan app Log Files for more information about working with log files.
Recommended solution
Linux
The sections below cover how to check and change the per-process open files limit for the CrashPlan app.
Step 1: Check the CrashPlan service open files limit
To check the open files limit in/proc/[PID]/limits, use the process ID of the CrashPlan service.
- Use
ps
to find the process ID:ps aux | grep crashplan
- Use
cat
to view the limit for the process ID:sudo cat /proc/[PID]/limits
In the following example, the CrashPlan service has a PID of 4698 and an open files limit of 1024 (shown in bold text).
code42@ubuntu:~$ ps aux | grep crashplan root 4698 10.5 5.0 821420 51572 pts/2 SNl 09:26 0:02 /usr/local/crashplan/jre/bin/java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx512m -Djava.net.preferIPv4Stack=true -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0 -Dc42.native.md5.enabled=false -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService code42@ubuntu:~$ sudo cat /proc/4698/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1024 1024 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 16382 16382 signals Max msgqueue size 819200 819200 bytes Max nice priority 20 20 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
Step 2: Increase the CrashPlan service open files limit
To increase the limit for the CrashPlan service, use the ulimit
command directly in the CrashPlan startup file, which is usually located in /usr/local/crashplan/bin/.
- Stop the CrashPlan service by running the following command:
sudo /usr/local/crashplan/bin/CrashPlanEngine stop
- Open the /usr/local/crashplan/bin/CrashPlanEngine file in a plain text editor.
- Add the following lines near the top of the file:
#Increase open files limit ulimit -n 65536
- Save and close the file.
- Run the following command to start the CrashPlan service:
sudo /usr/local/crashplan/bin/CrashPlanEngine start
In the following example, the lines in bold are what need to be added to the CrashPlan startup file.
#!/bin/bash ############################################################# # Init script for CrashPlanEngine ############################################################# #Increase open files limit ulimit -n 65536
OS X Yosemite version 10.10 and later
On OS X, the open file limits are governed by launchd
and sysctl
values.
- launchd: Processes are started by
launchd
, which imposes resource constraints on any process it launches. These limits can be retrieved and set using thelaunchctl
command (the default soft and hard values are 256 and unlimited, respectively). For OS X 10.7 and later, even though the default hard limit is "unlimited", you can't set the hard or soft limit to "unlimited" yourself. - sysctl: Operating system open files limits are set with
sysctl
. These limits can also impact running processes, so thelaunchd
andsysctl
open file limits should be set to the same values.
The sections below cover how to check and change these limits.
Step 1: Check the open files limits
Check the launchd and sysctl open files limits before you adjust them.
- Open the Terminal application.
- Check the launchd open files limit by running the following command:
sudo launchctl limit maxfiles
This command returns two values, a "soft" and a "hard" limit on each resource (example displayed below). When a process passes the "soft" limit it receives a signal from the operating system but isn't necessarily terminated. When it passes the "hard" limit it is immediately terminated.maxfiles 256 unlimited
- Check the sysctl open file limits by running the following command:
sudo sysctl -a | grep files
This command returns the kern.maxfiles and kern.maxfilesperproc limits (example displayed below).kern.maxfiles = 12288 kern.maxfilesperproc = 10240 kern.maxfiles: 12288 kern.maxfilesperproc: 10240 kern.num_files: 1521
Step 2: Increase the open files limit
Set the launchd
soft and hard limits to 65536.
- Open the Terminal application.
- Run the following command to set the soft and hard limits to 65536:
sudo launchctl limit maxfiles 65536 65536
- To make the new limits persist through system, you must create two configuration files:
- Create a /Library/LaunchDaemons/limit.maxfiles.plist file in a plain text editor and add the following lines:
<!--?xml version="1.0" encoding="UTF-8"?--> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>limit.maxfiles</string> <key>ProgramArguments</key> <array> <string>launchctl</string> <string>limit</string> <string>maxfiles</string> <string>65536</string> <string>65536</string> </array> <key>RunAtLoad</key> <true/> <key>ServiceIPC</key> <false/> </dict> </plist>
- Create a /Library/LaunchDaemons/limit.maxproc.plist file in a plain text editor and add the following lines:
<!--?xml version="1.0" encoding="UTF-8"?--> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>limit.maxproc</string> <key>ProgramArguments</key> <array> <string>launchctl</string> <string>limit</string> <string>maxproc</string> <string>2048</string> <string>2048</string> </array> <key>RunAtLoad</key> <true /> <key>ServiceIPC</key> <false /> </dict> </plist>
- Create a /Library/LaunchDaemons/limit.maxfiles.plist file in a plain text editor and add the following lines:
- Make sure the permissions and file/group ownership on this file are similar to those around it. Both plist files must have root permissions (
-rw-r--r--
). You can ensure that root permissions are in place by running the following commands:sudo chmod 644 /Library/LaunchDaemons/limit.maxfiles.plist sudo chmod 644 /Library/LaunchDaemons/limit.maxproc.plist
OS X Mavericks version 10.9 and earlier
On OS X, the open file limits are governed by launchd
and sysctl
values.
- launchd: Processes are started by
launchd
, which imposes resource constraints on any process it launches. These limits can be retrieved and set using thelaunchctl
command (the default soft and hard values are 256 and unlimited, respectively). For OS X 10.7 and later, even though the default hard limit is "unlimited", you can't set the hard or soft limit to "unlimited" yourself. - sysctl: Operating system open files limits are set with
sysctl
. These limits can also impact running processes, so thelaunchd
andsysctl
open file limits should be set to the same values.
The sections below cover how to check and change these limits.
Step 1: Check the open files limits
Check the launchd and sysctl open files limits before you adjust them.
- Open the Terminal application.
- Check the launchd open files limit by running the following command:
sudo launchctl limit maxfiles
This command returns two values, a "soft" and a "hard" limit on each resource (example displayed below). When a process passes the "soft" limit it receives a signal from the operating system but isn't necessarily terminated. When it passes the "hard" limit it is immediately terminated.maxfiles 256 unlimited
- Check the sysctl open file limits by running the following command:
sudo sysctl -a | grep files
This command returns the kern.maxfiles and kern.maxfilesperproc limits (example displayed below).kern.maxfiles = 12288 kern.maxfilesperproc = 10240 kern.maxfiles: 12288 kern.maxfilesperproc: 10240 kern.num_files: 1521
Step 2: Increase the launchd open files limit
Set the launchd
soft and hard limits to 65536.
- Open the Terminal application.
- Run the following command to set the soft and hard limits to 65536:
sudo launchctl limit maxfiles 65536 65536
- To make the new limits persist through system restarts, create or edit the /etc/launchd.conf file in a plain text editor and add the following line:
limit maxfiles 65536 65536
- Make sure the permissions and file/group ownership on this file are similar to those around it. You can also set these values on a per-user basis by editing or creating a file named $HOME/.launchd.conf. This can be useful if the CrashPlan app is installed "as user".
Step 3: Increase the sysctl open files limits
On OS X, the launchd
open file limit cannot exceed the sysctl
open file limits. Set both sysctl
open file limits to 65536.
- Open the Terminal application.
- Run the following commands to set the sysctl open files limits to 65536:
sudo sysctl -w kern.maxfiles=65536
sudo sysctl -w kern.maxfilesperproc=65536
- To make the new limits persist through system restarts, open /etc/sysctl.conf in a plain text editor and add the following lines:
kern.maxfiles=65536 kern.maxfilesperproc=65536