Splunk Completes Acquisition of Plumbr Learn more

File Attribute Operations

Checking the metadata for files in filesystem is a fast operation. However, when performing the checks like "is this a directory" or “can I write to this file” on large number of files, the accumulated duration for such checks escalates to a performance problem.
Identify bottlenecks in file operations with Plumbr.

How Plumbr will help you

The example above demonstrates a situation where during invocation of the FileSystemController.tree() service, 43 seconds have been spent on querying the attributes of different files. The most consuming of those was checking whether a file is actually a directory. It is quite self-evident what is happening here: we’re trying to render a filesystem tree, but the number of files is quite large, so we end up spending a lot of time on it.

Here is the offending code snippet: 
FileSystemTree descend(File root) {
  // ...
  if(root.isDirectory()) {
    File[] files = root.listFiles();
    if(files != null) {
      for (File child : files) {
        children.add(descend(child));
      }
    }
  }
  // ...
}

The Solution

There’s no easy way to make this work much faster if you actually need all of this information to be displayed. But if we look at the number of files, we could come up with a different idea. Will the user actually ever need the whole 564,206 files to be displayed in a tree at the same time? The answer is almost certainly no. So, implementing a lazy loading mechanism with fetching only the sub-trees that the user is interested in would dramatically improve the performance from the end user perspective.

Typically an individual folder does not contain all that many files on the same level, so we could limit the maximum depth of walking the file system:

FileSystemTree descend(File root, int maxDepth) {
  // ...
  if(maxDepth == 0) {
    // return
  }
  if(root.isDirectory()) {
    File[] files = root.listFiles();
    if(files != null) {
      for (File child : files) {
        children.add(descend(child, maxDepth - 1));
      }
    }
  }
  // ...
}