ZFS performance on FreeBSD

NOTE this is an ancient post. disregard everything it says.

For a combined application and database server with considerable per-database MySQL buffers and lots of synchronous IO on SATA drives:

• vfs.zfs.prefetch_disable="1" - disable prefetch, even on systems with decent amounts of RAM. With prefetch enabled the server freezes under moderate database activity. The associated arc_summary stats are mixed, with almost 100% miss for some reads.

  File-Level Prefetch: (HEALTHY)
  

  DMU Efficiency: 4.36b Hit Ratio: 82.85% 3.61b Miss Ratio: 17.15% 748.47m

      Colinear:                               748.47m
        Hit Ratio:                    0.02%   170.47k
        Miss Ratio:                   99.98%  748.30m
  
      Stride:                                 3.48b
        Hit Ratio:                    100.00% 3.48b
        Miss Ratio:                   0.00%   68.37k
  

An old mailing list posts suggest this is a "more or less known problem".

• vfs.zfs.arc_max="16G" - explicitly set a maximum amount of memory for ARC after working out what headroom is needed elsewhere. Otherwise ARC will cause pageouts for userland processes and it isn't as good as getting out of the way as it is supposed to be. Sometimes this is visible as THROTTLED in arc_summary, with a non-zero throttle count.

  ARC Summary: (THROTTLED)
      Storage pool Version:                   5000
      Filesystem Version:                     5
      Memory Throttle Count:                  8