Fetching a directory containing many files using the CLI

Content Software for File CLI Reference

Part Number

In order to fetch a directory that contains a large number of files, it is recommended to use the xargs command in a similar manner, as follows:

find -L <directory path> -type f | xargs -r -n512 -P64 weka fs tier fetch -v
Note: The pre-fetching of files does not guarantee that they will reside on the SSD until they are accessed.

In order to ensure that the fetch is effective, the following must be taken into account:

  • Free SSD Capacity: There has to be sufficient free SSD capacity to retain the filesystems that are to be fetched.
  • Tiering Policy: The tiering policy may release some of the files back to the object store after they have been fetched, or even during the fetch if it takes longer than expected. The policy should be long enough to allow for the fetch to complete and the data to be accessed before it is released again.