Milind Borate,
CTO, Druvaa

Mike is right; in-line deduplication is not limited to fixed-length blocking. Our Permabit Enterprise Archive product, for example, does variable-sized chunking for optimal deduplication.

You say:

Any deduped NAS/SAN device can-not control what data is flushed to it. The mounted file-system or storage driver flushes information which it may not make any sense to the device. In such cases the variable data-segment algorithm can’t be applied.

We cannot control when data is flushed to our device, but this does not mean that we cannot inspect the structure of the file as it is being written an make intelligent choices about where to set boundaries. Data being written sequentially is generally flushed sequentially, and in the case of out-of-order writes from the block cache we are able to do reassembly in memory or make guesses about file structure based on previously seen landmarks in the file. In our experience, we nearly always get deduplication as good as post-processing the file after it has been written entirely, and we do not introduce a dangerous “dedupe window” which can lead to falling far behind in the data stream.

This is technically more complicated to implement than post-process, but it can and has been done. There is more information about our deduplication technologies, which we call Scalable Data Reduction, on our website at http://www.permabit.com/products/sdr.asp.

Regards,
Jered Floyd
CTO, Permabit