In all the early releases, the motif for Druva has been Simplicity: simplicity for both end-users and for IT administrators. For end users, Druva inSync is now so simple to use that it just works without their knowing. It’s completely non-intrusive and works on all kinds of networks such as WAN and VPN. It’s so simple to use that end users can access their data from any Web browser without having to contact IT. Likewise, for IT administrators, it’s so simple to download (~40MB), install (under 20 minutes), and manage (almost zero maintenance) that the total cost of ownership is almost negligible. The simplicity motif has been a strong differentiator for Druva’s offerings.

For inSync 4.0, we made Storage and Bandwidth Optimization the motif. To optimize storage, we introduced App-aware Dedupe, an industry-first dedupe technology that offers a 90% storage savings across all user data and a 100% dedupe accuracy at the source (laptops) for supported applications such as Outlook and Office. To optimize bandwidth, we introduced the Octopus WAN optimization engine, a multi-threaded client architecture that does smart bandwidth throttling to offer a 5x performance gain for every client backing up on WAN.

The eye-catching red shack on the wharf (Rockport, Massachusetts) is often called Motif #1, a reference to its popularity among artists.

The theme for inSync 4.1 emerged naturally to “Scale” as customers were increasingly deploying Druva to more users in each of their environments. With release 4.1, we wanted to make inSync scale efficiently along several dimensions as outlined below –

Scale -

• 2000 users per server
• 16TB of data per server
• 200 parallel connections per server

Performance -

• We’re excited to introduce an innovative HyperCache technology, which can improve backup performance by 6x compared to inSync 4.0. HyperCache is an in-memory cache that can be configured to access the most optimal subset of your dedupe index in memory resulting in a high hit rate. The usual 80-20 rule applies here: with just a 30% subset of the dedupe index, Hypercache can deliver upwards of 75% hit rate. We recommend a 4GB of HyperCache size for every 1TB of data to maximize performance. The admin console offers a simple way for you to configure HyperCache for optimal performance.
• You can now configure an SSD storage for your dedupe index to further enhance your server performance. Lab results show a whopping 12x performance improvement with HyperCache and SSD configurations.
• You can now install Druva on a 64-bit system for enhanced performance.

Administration –

• 4.1 now supports a new administrative role in addition to a Server Administrator. A Profile Administrator role grants permissions to manage one or more user profiles in order to edit profile settings, add users, and manage data restore for those profiles. This is a great way to scale the administration tasks across your organization between server and profile administration.
• In light of the above role, we’ve enhanced our dashboard and reporting, so an administrator can get a customized view of their reports depending on their role.
• You can now automate the import of users to inSync from your Active Directory. A periodic import from your AD can be set up to dynamically add users to inSync.

Access -

• We’re very excited to announce mobile access of your data from iPads and iPhones. Check out our newest app at http://itunes.apple.com/app/insync/id420380654?mt=8#

We’re excited about the upcoming deployments of inSync 4.1 and the performance benefits to all of you. In my next blog, I’ll talk about the 2 editions of inSync 4.1 (Enterprise and Professional), how they compare, and which one is right for you. Stay tuned….

The idea of “app-aware deduplication” emerged from the fact that complex applications like MS Outlook or Exchange need much more intelligent deduplicate removal than simple block based approach.

Each data block in PST is of fixed size and usually contains a header and a footer (ref: libpst ) which makes it impossible for simple dedupe approaches to identify block boundaries and hence restricting deduplication accuracy to just 30-40%.

Application aware data deduplication depends upon APIs exposed by the application to understand the construct of on-disk data and deduplicate at the logical-block or message level. This guarantees 100% deduplication accuracy and faster processing of data.

Another interesting change is shift from PostgreSQL database to no-SQL Oracle embedded database. This small (less than 1MB in size) embedded database removes the heavy “SQL” and networking layer between the server and database, hence greatly improving performance and scalability. The new engine can now support 16TB of dedupe data and about 200 parallel backups.

In a nutshell, the Blackbird engine will have the following features -

1. App-Aware deduplication
2. Light-weight and highly scalable
3. Simple to install and zero-maintenance
4. Near-CDP – timeline/event based near-continuous backups
5. Search enabled restores
6. Replication (to be showcased soon

InSync v4.0
InSync v4 is definitely a new benchmark for laptop backup. I am extremely confident that if anyone tries this solution will never buy anything else for laptop backup. With new storage, redesigned WAN Optimization and dashboard, its clearly leaps and bounds ahead of what’s available in the market.

More about new features – http://www.druva.com/insync/version-4-0
Download inSync v4 – http://www.druva.com/download/insync

So, to make the brand recall simpler, we spent some good time, effort and money to removed the duplicate “A”.

The website has already been migrated (see www.druva.com)and now has a new cool logo as well. All the email addresses will be carried forward as-they-are to the new domain, and older email addresses will still be valid. Request you to make a change in your address book as well.

We soon will be migrating other sub-domains (blog, kb, forums etc.). The changes may take some time, and request you to be patient.

If you are a customer or listed somewhere in our salesforce.com, an email is probably already waiting in your Inbox guiding you through changes

The entire team has been super busy to make this happen. And I am sure it would be quite evident when you give it a try.

Reinventing Backup

Phoenix is designed ground-up for remote backups. Here are some of the key product features which make it ultra special -

1. Global Source Based Data Deduplication – Over 90% reduction in backup time, bandwidth and storage.
2. WAN Optimization – Understands high latency and noisy networks.
3. Near Continuous Data Protection – snapshot/restore-points based point-in-time restores. No age-old full, incremental backups.
4. Smart Bandwidth Scheduling - Set smart bandwidth limits for each backup schedule.

The Road Ahead

What we currently have is just a platform which will be used to showcase some market changing features -

1. Search Based Restore – We missed this feature in v1.0, but should be available in the next v1.2 release
2. “Blackbird SR-71“ – A new storage engine with application aware data deduplication. This should be able to match an attachment inside exchange store at New Jersey to a file stored in a file-server at Kent. This should set the standards for backup performance.
3. Long Distance Replication – Replicate backed up data over noisy long distance IP networks.
4. Advanced Dashboard – The second best reporting dashboard (after Google Analytics).

Application aware Agents - Phoenix currently only comes with generic Windows agent, we plan to introduce these starting v2.0

Useful Links -

• Product Page - www.druvaa.com/phoenix/network-backup
• Download - www.druvaa.com/download/phoenix
• Quick Setup Guide - http://www.druvaa.com/phoenix/quick-setup-guide

I welcome you guys to download a copy and share your feedback !

In Short -
We are working on a new storage engine codename Blackbird (based on the SR-71 legend). The new engine will use application specific deduplication technology to improve performance and bandwidth/storage savings.

Initially planned for inSync v3.1 and Phoenix v1.0 , this now will be available in next major releases.

The longer version -
For the past two years, we have been doing experiments on various different algorithms for global source based data deduplication. While releasing inSync v2.0 we finalized on chunk based or variable-block based data deduplication, because of the simple fact that it was tough to find similar data blocks at natural block boundaries across different users. We also worked on the performance which gradually improved over time.

While the approach was reasonably accurate, there was a scope of significant improvement. We realized that 90% of the backup data on customer PCs comes from the documents and PST files, hence something totally focussed on PST files can dramatically improve the deduplication performance.

Also, while working on Phoenix, we came across a bigger challenge of finding duplicates across different data sources within the enterprise. We soon realized that simple block based approach will not take us too far. We also realized that most of the vendors use fixed and variable block/chunk based hashing techniques. This works well for them, because they have been treating backups as “byte streams”, and the only way to remove duplicates is fixed or variable size data deduplication.

Looking at various data types and possible ways improve, we could clearly see two fundamental changes in our approach which could bring paradigm shift in data deduplication -

1. For accuracy – Application aware data deduplication
2. For performance – Hierarchical block based deduplication

Application aware deduplication, can actually pin point duplicates across PST file attachments and  normal office documents.

On the PC side, majority of the data is office documents and Email files. This makes it simpler to introduce the new approach, but still a lot of work needs to be done to productise it. For Phoenix, the problem is much bigger and would take some more time to solve.

The new engine should be ready soon. It would be shipped first in inSync v4.0 early next year and then in Phoenix v2.0 . In the next few posts, I will try and get some benchmark data.

Definition

Data deduplication or Single Instancing essentially refers to the elimination of redundant data. In the deduplication process, duplicate data is deleted, leaving only one copy (single instance) of the data to be stored. However, indexing of all data is still retained should that data ever be required.

Example
A typical email system might contain 100 instances of the same 1 MB file attachment. If the email platform is backed up or archived, all 100 instances are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; each subsequent instance is just referenced back to the one saved copy reducing storage and bandwidth demand to only 1 MB.

Technological Classification

The practical benefits of this technology depend upon various factors like –

1. Point of Application – Source Vs Target
2. Time of Application – Inline vs Post-Process
3. Granularity – File vs Sub-File level
4. Algorithm – Fixed size blocks Vs Variable length data segments

A simple relation between these factors can be explained using the diagram below -

Target Vs Source based Deduplication

Target based deduplication acts on the target data storage media. In this case the client is unmodified and not aware of any deduplication. The deduplication engine can embedded in the hardware array, which can be used as NAS/SAN device with deduplication capabilities. Alternatively it can also be offered as an independent software or hardware appliance which acts as intermediary between backup server and storage arrays. In both cases it improves only the storage utilization.

On the contrary Source based deduplication acts on the data at the source before it’s moved. A deduplication aware backup agent is installed on the client which backs up only unique data. The result is improved bandwidth and storage utilization. But, this imposes additional computational load on the backup client.

Inline Vs Post-process Deduplication

In target based deduplication, the deduplication engine can either process data for duplicates in real time (i.e. as and when its send to target) or after its been stored in the target storage.

The former is called inline deduplication. The obvious advantages are -

1. Increase in overall efficiency as data is only passed and processed once
2. The processed data is instantaneously available for post storage processes like recovery and replication reducing the RPO and RTO window.

the disadvantages are -

1. Decrease in write throughput
2. Extent of deduplication is less – Only fixed-length block deduplication approach can be use

The inline deduplication only processed incoming raw blocks and does not have any knowledge of the files or file-structure. This forces it to use the fixed-length block approach (discussed in details later).

The post-process deduplication asynchronously acts on the stored data. And has an exact opposite effect on advantages and disadvantages of the inline deduplication listed above.

File vs Sub-file Level Deduplication

The duplicate removal algorithm can be applied on full file or sub-file levels. Full file level duplicates can be easily eliminated by calculating single checksum of the complete file data and comparing it against existing checksums of already backed up files. It’s simple and fast, but the extent of deduplication is very less, as it does not address the problem of duplicate content found inside different files or data-sets (e.g. emails).

The sub-file level deduplication technique breaks the file into smaller fixed or variable size blocks, and then uses standard hash based algorithm to find similar blocks.

Fixed-Length Blocks v/s Variable-Length Data Segments

Fixed-length block approach, as the name suggests, divides the files into fixed size length blocks and uses simple checksum (MD5/SHA etc.) based approach to find duplicates. Although it’s possible to look for repeated blocks, the approach provides very limited effectiveness. The reason is that the primary opportunity for data reduction is in finding duplicate blocks in two transmitted datasets that are made up mostly – but not completely – of the same data segments.

For example, similar data blocks may be present at different offsets in two different datasets. In other words the block boundary of similar data may be different. This is very common when some bytes are inserted in a file, and when the changed file processes again and divides into fixed-length blocks, all blocks appear to have changed.

Therefore, two datasets with a small amount of difference are likely to have very few identical fixed length blocks.

Variable-Length Data Segment technology divides the data stream into variable length data segments using a methodology that can find the same block boundaries in different locations and contexts. This allows the boundaries to “float” within the data stream so that changes in one part of the dataset have little or no impact on the boundaries in other locations of the dataset.

ROI Benefits

Each organization has a capacity to generate data. The extent of savings depends upon – but not directly proportional to – the number of applications or end users generating data. Overall the deduplication savings depend upon following parameters –

1. No. of applications or end users generating data
2. Total data
3. Daily change in data
4. Type of data (emails/ documents/ media etc.)
5. Backup policy (weekly-full – daily-incremental or daily-full)
6. Retention period (90 days, 1 year etc.)
7. Deduplication technology in place

The actual benefits of deduplication are realized once the same dataset is processed multiple times over a span of time for weekly/daily backups. This is especially true for variable length data segment technology which has a much better capability for dealing with arbitrary byte insertions.

Numbers

The dedupication ratio increases everytime to pass the same complete data-set through the deduplication engine.

If compared against daily full backups, which I think is not widely used today, the ratios are close to 1:300.  Most if the venders use this as a marketing jargon to attract customers, even though none of their customers could be doing daily full-backup

If compared against modern day incremental backups, our customer statistics show that, the results are between 1:4 to 1:50 for source based deduplication.