Endpoint data is more dispersed and diverse than ever before. Some of our customers are protecting up to 1PB of data with inSync, and one of the key requests we get is to have the tools understand “what” they are backing up and how can they defuse this rapidly expanding data.

This new addition is designed to answer most of these questions with the following features :

• Data Analytics: This lets the admin gain insight into the composition of the backed up data and understand the key trends fueling the growth of data.
• Global Search: Searches the meta-data to find files with certain keywords across the users. It also helps the admin gain access to the data refereed in the results
• Reports: Helps schedule search and analytics and generate custom reports
• APIs & Tools: APIs and different export format to integrate with any existing software

Essentially this is an eDiscoveryLite for enterprise endpoints, and yet again demonstrates our commitment to enterprise endpoints, an area we feel is extremely important but lacking in tools and support.

So far, the feedback from the Data Insights beta has been great. Interestingly, some customers came back with a different use case and requested if this could be combined with our SafePoint product (data loss prevention) or with data sharing, to help the admins better understand the propagation of data and how to secure it.

You can learn more about the product, which is currently free for existing customers, by going to the Advanced Data Insights page.

Although it did affect Netflix, foursquare and few others, thankfully Druva cloud services were completely unaffected by this. Here is a small note on how we managed to keep our promised SLA.

I think its plain ignorance or mis-planing to assume 100% availability of underlying infrastructure. Just like any hardware, the AWS infrastructure is prone to failures, but the knowledge of these potential failure points can help improve availability.

Since its a backup service, we have have divided our cloud design into 3 parts based on the availability and durability guarantees :

• Config (most available): Configuration data stored in Amazon RDS
• Meta-Data : Druva Dedupe file-system spanning across Cassandra nodes
• Data (most durable): Stored in S3

And some design changes we incorporated to avoid downtimes :

• Multi-Zone replication: Both RDS and Cassandra nodes are replicated across 3 availability zones. We use Cassandra in full-consistency mode and heavily rely on its self-healing, in case of service failures.
• Reduced Dependency on EBS: EBS is a software abstraction of an underlying SAN storage. And two independent EC2 instances may share same SAN for EBS. Given this we shifted our focus from EBS to local-storage for meta-data.
• Extra space copies in S3: We so maintain some extra redundancy on top of S3 for most referenced blocks. This essentially is to avoid the random (but less frequent) S3 time-outs and improve durability of most concurrent data.

We surely paid more for improved availability, but there are simple design changes which can help save as well. For example the 3-way replication increased our compute(EC2) cost by over 200%, but because of extra spare we could increased the data stored per instance, which was earlier restricted to maintain a good cache-vs-on-disk ratio.

Time and again multiple enterprise customers, especially those who are migrating from competing solutions, ask us about scalability of Druva inSync. Since the launch of v4.0, inSync has scaled exceptionally well, especially for large deployments. The software has succeeded where majority of competing solutions have failed or turned off deduplication.

About a week back, (on request of a large customer) we started testing one of the competing solutions. We tested the software for 1 million files of total size of 2TB, of which 48% was duplicate. Insync finished the backup in about 22 hours and the competing software is still backing up.

InSync doesn’t support any “integration” with deduplication, but the whole software was designed around the deduplication and CDP. There is NO flag to turn off dedupe and there never will be.

This article focuses on my thoughts on how Druva succeeds where majority of others fail.

Why Source Deduplication Fails to Scale for Majority Vendors ?
The biggest bottleneck for performance scalability of deduplication is the random disk IO performance. Almost all dedupe systems include a database to store the block-hash index which needs to be checked for every hash check. A server class magnetic disk usually offers a latency of 8-12ms which restricts the hash matches to about 100/sec, throttling the dedupe performance drastically.

Now, when the data set is small the entire index can reside in memory and hence the hash checks as much faster. As the index grows, the I/O congestion brings down the software’s capacity to perform inline deduplication.
Consider this: Just about 1000 users can create over 10 Billion blocks for backup. And checking them with a rate of 100/sec could take 3.21 years.

Learnings from Storage Guys
Data domain had an interesting approach. They optimized their inline dedupe performance for backup streams. Since the backup was mostly for servers with few large files and the data streams were mostly long streams of data in tar format, Data domain used a simple index read-ahead algorithm to load the relevant parts of the index before the stream blocks hashes reached the server. Since the streams changed less than 10% across two simultaneous backups, the algorithm helped deduplicate them at a very fast pace.

Solid State Disks
A simple solution to the random-I/O problem is using SSDs to store the index. Although we did tweaked/changed certain features to support SSDs but the solution wasn’t complete because of the size limitation imposed by them.

Two Step Approach for Druva: No-SQL + HyperCache
The “Data Domain approach” did not work for us as our data was much more random and coming from different sources. But on the flip side we had much more knowledge of the data formats we were backing up.
The first step towards scalability was to get rid of the inbuilt SQL database which imposed a lot of latency because of SQL query serialization and execution. We replaced PostgreSQL with Oracle no-SQL BDB as an embedded database, which improved the performance and much simpler to maintain.

The second major innovation was HyperCache – a selective in-memory cache of index. Hypercache constitutes of both a positive and a negative cache, which remembers and caches both the most probable and the least probable hashes for on-going backup. HyperCache uses an ever learning algorithm and uses different parameters like time, frequency and probability of a hash to cache it.

The Result
The result was 85% reduction in disk I/O by using 4GB of RAM for every 1TB of data stored. The reduction in IO translates to 4X better scalability, and the solution can easily scale to thousands of users with linear improvement in scalability/performance.

Use of SSDs further improves the performance by 6X. InSync core has been modified to keep only the most concurrent part of the database index on SSDs and optimize it for solid state drives.

I’ve been travelling pretty heavily over the last 2 months, and had an opportunity to meet and learn from a lot of customers. The best companion during this travel has been my iPad. Here’s my small list of productivity apps/accessories which I use the most:

• inSync iPad app – lets me access all files and all the versions backed up from my laptop
• Smart cover – makes typing emails simple
• Easy Sign app – makes signing documents easy (I realized this was the only reason I used my fax/scanner)
• Kindle app – Bought 2 books: The idea book & The upside of irrationality
• iPad VGA adapter – allows me to present directly from my iPad
• Evernote – for taking meeting notes

I’m currently a happy beta tester of the new version of the inSync app for iPad which allows offline access of folders marked as “favorites”, and this was an absolute life saver when I was travelling in Europe and didn’t want to pay heavy roaming charges

I see tablets as great access devices, which in my opinion would affect two markets the most : desktop and printer. Although I am still adjusting to carrying 1 extra device, but IMO the laptop may soon become more of an in-office/limited-mobility device.

When we started the journey about 3 years ago, we were almost the only startup focused on Enterprise Laptop Backup and the only case study for us to follow was that of Connected (which was acquired by Iron Mountain and then eventually left to die). We have come a long way and have 600+ strong customers with some great names like PWC, NASA, Xerox, Schlumberger etc. When you have a fire at your home,  all you care is a good/reliable fire extinguisher and value the brand less, and I believe that’s what happened with us with all these great brands.

And now its good to see many other players joining the game. Last week we saw announcements by i365 Cloud (Seagate) and CommVault. Mozy and Carbonite have also done a good job in educating the consumer markets which is eventually spreading awareness in businesses as well.

With the February launch of Druva inSyne Enterprise and last week’s Backup Cloud announcement, we have yet again established our leadership in innovation.

Our salesforce stats reveal that about 63% of qualified leads eventually buy InSync and we almost never lose to our competitors (we lost just 8 against 320 opportunities in last 1 year). We’re super confident of improving this track record with upcoming customer reach programs and product announcements. Stay tuned.

Enter a Hack Day event. The Hack Day event provides the perfect opportunity for passionate employees to block times on their calendars, bring their favorite idea to the forefront, form teams with like-minded employees within the organization, and just hack away. I’ve seen this concept work like magic in companies like Yahoo and Intuit. Today, we’re having our first Hack Day event at Druva!

A Hack Day Event

The Druva Hack Day, a 24-hour code marathon, began today at 9AM. The participation from engineers and QA has been almost at a 100%, which is amazing! We have 19 teams (1-3 people) working on their passionate ideas as we speak. We also have our fearless leader, Jaspreet, hacking his favorite idea. Common sights as you walk along the offices are huddles, whiteboard discussions, intense coding, and camaraderie, all of which one would expect to find in a successful hack day event.

We have the final presentations scheduled tomorrow morning at the end of the 24-hour event, so everyone can share their latest and greatest creations with the rest. Exciting awards await the winners, whose works will be judged based on their creativity, technical difficulty, and most of all, the impact in the market and on our customers. Can’t wait to see what the teams have in store!

In all the early releases, the motif for Druva has been Simplicity: simplicity for both end-users and for IT administrators. For end users, Druva inSync is now so simple to use that it just works without their knowing. It’s completely non-intrusive and works on all kinds of networks such as WAN and VPN. It’s so simple to use that end users can access their data from any Web browser without having to contact IT. Likewise, for IT administrators, it’s so simple to download (~40MB), install (under 20 minutes), and manage (almost zero maintenance) that the total cost of ownership is almost negligible. The simplicity motif has been a strong differentiator for Druva’s offerings.

For inSync 4.0, we made Storage and Bandwidth Optimization the motif. To optimize storage, we introduced App-aware Dedupe, an industry-first dedupe technology that offers a 90% storage savings across all user data and a 100% dedupe accuracy at the source (laptops) for supported applications such as Outlook and Office. To optimize bandwidth, we introduced the Octopus WAN optimization engine, a multi-threaded client architecture that does smart bandwidth throttling to offer a 5x performance gain for every client backing up on WAN.

The eye-catching red shack on the wharf (Rockport, Massachusetts) is often called Motif #1, a reference to its popularity among artists.

The theme for inSync 4.1 emerged naturally to “Scale” as customers were increasingly deploying Druva to more users in each of their environments. With release 4.1, we wanted to make inSync scale efficiently along several dimensions as outlined below –

Scale -

• 2000 users per server
• 16TB of data per server
• 200 parallel connections per server

Performance -

• We’re excited to introduce an innovative HyperCache technology, which can improve backup performance by 6x compared to inSync 4.0. HyperCache is an in-memory cache that can be configured to access the most optimal subset of your dedupe index in memory resulting in a high hit rate. The usual 80-20 rule applies here: with just a 30% subset of the dedupe index, Hypercache can deliver upwards of 75% hit rate. We recommend a 4GB of HyperCache size for every 1TB of data to maximize performance. The admin console offers a simple way for you to configure HyperCache for optimal performance.
• You can now configure an SSD storage for your dedupe index to further enhance your server performance. Lab results show a whopping 12x performance improvement with HyperCache and SSD configurations.
• You can now install Druva on a 64-bit system for enhanced performance.

Administration –

• 4.1 now supports a new administrative role in addition to a Server Administrator. A Profile Administrator role grants permissions to manage one or more user profiles in order to edit profile settings, add users, and manage data restore for those profiles. This is a great way to scale the administration tasks across your organization between server and profile administration.
• In light of the above role, we’ve enhanced our dashboard and reporting, so an administrator can get a customized view of their reports depending on their role.
• You can now automate the import of users to inSync from your Active Directory. A periodic import from your AD can be set up to dynamically add users to inSync.

Access -

• We’re very excited to announce mobile access of your data from iPads and iPhones. Check out our newest app at http://itunes.apple.com/app/insync/id420380654?mt=8#

We’re excited about the upcoming deployments of inSync 4.1 and the performance benefits to all of you. In my next blog, I’ll talk about the 2 editions of inSync 4.1 (Enterprise and Professional), how they compare, and which one is right for you. Stay tuned….

Yes, I said calf, or rather, Tom did.

But this is the great thing about bloggers and writers, and presenting to your peers. They look at things differently, offer different opinions and insights. And to Tom, the rise that Druva has had, amassing over 600 customers globally in 3 short years, was nothing short of miraculous, reminding him of how a calf can stand on it’s own a few short hours after being born. I’d never really thought of it that way, but after reading it, felt it was a fitting comparison. It means we must be addressing an important problem, and the way we’re addressing it, is in the right way.

Of course, to piggy back off of this, the question now is, can Druva grow from a calf to a bull. We’d like to think so, – the recent press coverage we’ve received on our launch was pretty compelling, picked up by InfoStor, StorageNewsletter.com and NetworkWorld among others. In an article on Network Computing Howard Marks said:

“I am constantly amazed at how frequently organizations leave the data on those laptops essentially unprotected. By taking a unique application aware approach to data deduplication, Druva’s Insync may be the best solution to date to this seemingly intractable problem”

Bullish words indeed…

Druva Software is one of the Entrepreneur Magazine’s Top 25 Fast Growth Companies and, one of the Top 6 Companies to Watch out For.  The list has been put together by the AII World Network. Click here to read the report.

Druva Software is one of Business World’s Change Agents featured in this month’s edition. The article features 10 emerging companies with the potential to change the contours of the Indian Infotech Industry. Click here to read the complete article.

Says Manek, writing for ZDNet: “… Data deduplication and WAN optimisation have been well-established technologies in the storage and enterprise backup…  But this is the first time that they’ve been brought together in a laptop backup system. …if it works, it could be all what you need.”

Click here for complete article.

Caroline Donnelly writes for CRN: “If you lose the data stored on laptops and servers, your business is pretty    much ruined… (inSync) significantly reduces the bandwidth and storage required to back up data on laptops.”

Click here for complete article.

Jean-Jacques Maleval The Storage Newsletter features Druva commencing operations in the UK with the release of inSync  v4.0  saying: “…inSync’s application-aware deduplication approach makes the backup process more efficient and  cost- effective by significantly reducing storage bandwidth and capacity needs.”  Click here for complete article.

Chris Mellor, writing for The Register on inSync: “The software sniffs or checks what type of network connection ….and optimizes the style of network transmission, with smart bandwidth throttling, so as to minimize the network burden.”  Click here for complete article.

Steve Evans for CBR writes of inSync: “…faster and less obtrusive than other products on the market.” Click here for complete article.

Says Claus Egge, writing of Druva inSync v4.0 on his blog: “Mobility has complicated backup and restore for many years. By addressing the pain points of bandwidth and performance (these) mobile computers finally stand a chance of getting proper protection with Druva.”  Click here to read Claus Egge’s blog.

And what I saw totally changed my perception of  ”fast” and “small”. Forever.

If you have missed my earlier posts, Blackbird (name inspired from the SR71-Blackbird) is a new storage engine for inSync and Phoenix which will introduce Invent “Application Aware Data Deduplication” and get rid of any dependency on a SQL database.

Getting rid of SQL database shrinks the Blackbird core to only about 2 MB against current 32MB. It also avoids the overhead of TCP connects to SQL DB and SQL query optimizer, making it blazingly fast and light weight.

Application aware deduplication makes sure data deduplication happens at the logical level. For example, rather than treating a PST file like a stream of blocks, we would understand and deduplicate message by message. This guarantee faster processing and 100% deduplication accuracy.

Quickly highlighting the features from today’s presentation -

1. Fast - Amazing speed.
2. Light-weight - Multi-threaded. No SQL Database.
3. Scalable – High level of fault tolerance against data corruptions in the store.
4. Small - InSync v4.0 will probably shrink to just 12MB !!
5. Simple – No DB required for installation !

The public beta for InSync v4.0 is expected by end of April. I am extremely eager to showcase this engineering marvel.

Will follow with more posts and some screenshots !