Virtualization has been a key driver behind every major trend in software, from search to social networks to SaaS, over the past decade. In fact, most of the applications we use — and cloud computing as we know it today — would not have been possible without the server utilization and cost savings that resulted from virtualization.

But now, new cloud architectures are reimagining the entire data center. Virtualization as we know it can no longer keep up.

As data centers transform, the core insight behind virtualization — that of carving up a large, expensive server into several virtual machines — is being turned on its head. Instead of divvying the resources of individual servers, large numbers of servers areaggregated into a single warehouse-scale (though still virtual!) “computer” to run highly distributed applications.

Every IT organization and developer will be affected by these changes, especially as scaling demands increase and applications get more complex every day. How can companies that have already invested in the current paradigm of virtualization understand the shift? What’s driving it? And what happens next? MORE

Without a file system, modern computers would not operate. At some point, all I/O and most data on a computer finds its way into a file system — making it one of the most indispensable system software components.

File systems have always been designed for a disk-drive-centric storage environment. It’s a fundamental paradigm that has existed since the beginning of computing. But the world is changing. As we confront unprecedented amounts of data, much of it in real-time, there is a shift from disk-centric to memory-centric computing. We can reliably predict, with the cost of system memory decreasing, that memory for both storage and compute will be the exact same thing.

This shift to memory-centric computing requires an entirely new file and storage system. And that’s where Tachyon comes in to play. MORE

The history of computing can be largely described by architectural eras demarcated by the near-continuous ebb and flow from centralized to distributed computing. The first generation was centralized, with mainframes and dumb terminals defining the era. All computing was done centrally by the mainframe with the terminal merely displaying the resulting operations.

As endpoints (and networks) became more capable with additional processing and storage, the generation of client-server computing took hold. This architecture leveraged both endpoint and central capacity, giving users the benefit of hi-fidelity applications that communicated to centralized data stores in a seamless (or so intended) fashion. Unlocking less expensive and available compute at the endpoint unleashed an entire generation of new applications and businesses such as Facebook, Twitter, Square and many others.

Over the past decade, cloud computing and software as a service (SAAS) have moved the needle back once again toward a centralized architecture. Processing is centralized in the cloud datacenter and endpoints simply display the resulting operations, albeit in a more colorful way than their simple terminal predecessors. This is now changing.

Our mobile devices have become supercomputers in our hand. The processing power and storage capacity of these devices are now 100x more capable than PCs of 20 years ago. History has shown that as processing becomes available, new applications and architectures happily utilize the excess capacity. Enter the new world of cloud-client computing where applications and compute services are executed in a balanced and synchronized fashion between your mobile endpoint and the cloud.

Because smartphones are such beefy computers, developers have been rushing to take advantage of the available computing horsepower. Until now, this mostly meant writing native applications using Apple’s XCode or Eclipse/ADT for Android. But native apps are a pain: they typically require separate front-end engineers, there is little code shared between apps, and there is no concept of coordination with the back-end (cloud) services. All of this work must be handcrafted on a per app and per OS basis, rendering it costly and error-prone. It’s a duct tape and bailing twine approach for delivering a marginally better user experience.

That is, until Meteor. The Meteor platform brings back the goodness of the Web without sacrifice. Teams can share code across Web and mobile front ends, build super slick apps with a single code base across Android and iOS, and utilize a framework for integrating front-end and cloud operations. For the first time, there is a simple and elegant way to create applications that leverage the best of the client and the cloud, yielding applications that are high-fidelity and synchronized with the most advanced back-end/cloud services.

Meteor delivers dramatic productivity improvements for developers who need to deliver great experiences across Web, iOS, Android and other mobile platforms and enables the computational “oomph” available on smartphones to do more than just render HTML. Meteor delights users with Web and app experiences that are fluid and fast.

Meteor has the technology to usher in the new world of cloud-client computing and we couldn’t be more proud to be investors in the team that makes all of this happen.

 

A while back I wrote a blog post suggesting that datacenter infrastructure would move from an on-premise operation to the cloud. It may have seemed counter-intuitive that the infrastructure itself would become available from the cloud, but that’s exactly what’s happening.

We’ve now seen everything from security to system management to storage evolve into as-a-service datacenter offerings, yielding all the benefits of SaaS — rapid innovation, pay-as-you-go, no hardware installation — while at the same time providing rich enterprise functionality.

As the datacenter gets dis-intermediated with the as-a-service paradigm, an interesting opportunity exists for the “big data” layer to move to the cloud. While big data is one of the newer parts of the infrastructure stack — and should have been architected and delivered as a service from the start — an estimated 90+% of Fortune 2000 companies carry out their big data analytics on-premise.  These on-premise deployments are complex, hard to implement, and have already become something of a boat anchor when it comes to attempts to speed up big data analytics. They perfectly define the term “big drag.”

Without question the time has come to move big data to the cloud and deliver this part of the infrastructure stack as a service. Enter Cazena — our latest investment in the big data sector. The Cazena founders were former leaders at Netezza, the big data appliance leader that went public and was acquired by IBM for $1.7 billion. Prat Moghe, founder & CEO of Cazena, previously led strategy, product and marketing at Netezza. Prat has teamed up with Jit Saxena, co-founder of Netezza, and Jim Baum, the CEO of Netezza — all leaders in the big data industry.

This team knows a great deal about big data and agility of deployment. Ten years ago (long before the term big data was being used), the Netezza team came up with a radically simple big data appliance. Appliances reduced the sheer complexity of data warehouse projects — the amount of time and resources it took to deploy and implement big data.

In the next decade, even faster deployment cycles will be required as businesses want data on-demand. Additionally, the consumption pattern has changed as the newer data stack built using Hadoop and Spark has broadened the use of data. A new cloud-based, service-oriented deployment model will be required. The Cazena team is uniquely positioned to make this a reality.

We could not be more thrilled to be backing the team that has the domain expertise and thought leadership to change the face of big data deployments. Big data is changing the way the world processes information, and Cazena is uniquely positioned to accelerate these efforts.

I was introduced to Paula Long the CEO of DataGravity about the same time I arrived at a16z (nearly four years ago).  Every time a new storage deal was pitched to us, I would call Paula to get her thoughts. Given my own background in storage and systems software, I was blown away at Paula’s depth and knowledge in the space. Not only did she articulate every technical nuance of the project we discussed, she had an uncanny feel for what was likely to happen in the future.

Paula casually rattled off every company doing similar things, price and performance of solid-state storage, file systems, volume managers, device drivers, block interfaces, meta data, NAS, SAN, objects, and security. It was enough to make my head spin, yet she analyzed every situation with a clarity that I had never seen before. I had known Paula as the founder of EqualLogic (her prior storage company acquired by Dell for $1.4 billion in 2008), but her insight and wisdom about everything storage far exceeded that of anyone I had met. When she came to me with her own ideas for a new storage company there was no hesitation. Betting on Paula would result in something really special. In December 2012 we invested in DataGravity.

When we talked about DataGravity in those days, Paula would tell me how the real future of storage was unlocking the information residing in the gazillions of files and terabytes of unstructured data that organizations store but never use. She articulated that most other storage companies were in a race to zero; chasing the faster and cheaper angle, with their solid-state storage and incremental innovation. “Table stakes,” she would say. “DataGravity is going to do something never done before. We are going to unlock the value of storage. Storage is the obvious place for intelligence to be surfaced.” This all sounded great, but – even with my background in the space – I never fully appreciated what Paula had envisioned. She had a secret.

Today, DataGravity is unveiling the world’s first data-aware storage system. The system is quite simply revolutionary. We saw a demonstration of the system’s capability at a board meeting a few months ago, and that is when it all came together for me. This was not some incremental system that everyone else was building, but an entirely new way of managing storage and information. I left the board meeting thinking that all storage systems in the future would have elements of the DataGravity concepts. It was truly new thinking.

This was not some incremental system that everyone else was building, but an entirely new way of managing storage and information.

The secret sauce DataGravity brings to the market is making dumb storage smart, all in a single system. DataGravity is both a primary storage array and an analytics system combined into one. The combination — without any performance or operational penalty — means, for the first time, that organizations can use their primary storage for file storage, IT operations, AND analytics at the point of storage. “Data-aware” means indexing and giving storage intelligence before it is stored. Instead of having dedicated and expensive secondary systems for analytics, operations and data analysis, DataGravity does it all in one place.

DataGravity is about to change the way we think about storage. From the demographics of data, to data security, to searching and trend information, the system will unlock an entire class of capabilities that we have not yet begun to comprehend. For example, imagine knowing when a file is being written or corrupted, before it is accessed. Or being able to identify subject-matter experts in an organization based on who is writing the most content on what and when. Or determining data ownership and control and correlate this with active or inactive employees. All this from a “storage” system.

So here we are today at an amazing inflection point in the history of storage. Twenty years from now, we’ll look back at this day as the day storage went from being dumb to being smart. The day that transformed the way the world stores its information. Just as Paula predicted, and just as Paula knew.

 

 

The mobile revolution has spread beyond the mini supercomputers in our hands all the way to the datacenter.

With our expanded use of smartphones comes increased pressure on servers to help drive these devices: The activity we see everyday on our phones is a mere pinhole view into all that’s happening behind the scenes, in the massive cloud infrastructure powering all those apps, photo-shares, messages, notifications, tweets, emails, and more. Add in the billions of devices coming online through the Internet of Things — which scales through number of new endpoints, not just number of users — and you begin to see why the old model of datacenters built around PCs is outdated. We need more power. And our old models for datacenters are simply not enough.

That’s where mobile isn’t just pressuring, but actually changing the shape of the datacenter — displacing incumbents and creating new opportunities for startups along the way. READ MORE

The promise of big data has ushered in an era of data intelligence. From machine data to human thought streams, we are now collecting more data each day, so much that 90% of the data in the world today has been created in the last two years alone. In fact, every day, we create 2.5 quintillion bytes of data — by some estimates that’s one new Google every four days, and the rate is only increasing. Our desire to use, interact, and learn from this data will become increasingly important and strategic to businesses and society as a whole.

Yet, while we are collecting and storing massive amounts of data, our ability to analyze and make use of the data is stuck in information hell. Even our most modern tools reflect an older, batch-oriented era, that relies on queries and specialized programs to extract information. The results are slow, complex and time consuming processes that struggle to keep up with an ever-increasing corpus of data. Quite often, answers to our queries are long outdated before the system completes the task. While this may sound like a problem of 1970s mainframes and spinning tape, this is exactly how things work in even the most modern Hadoop environments of today.

More data means more insight, better decisions, better cures, better security, better predictions — but requires re-thinking last generation tools, architectures, and processes. The “holy grail” will allow all people or programs to fluidly interact with their data in an easy, real-time, interactive format — similar to a Facebook Search or Google Search engine. Information must become a seamless and fundamental property of all systems, yielding new insights by learning from the knowns and predicting the unknowns.

That’s why we’re investing in Adatao, which is on the leading edge of this transformation by combining big compute and big data under one beautiful document user interface. This combination offers a remarkable system that sifts through massive amounts of data, aggregating and machine-learning, while hiding the complexities and helping all users, for the first time, to deal with big data analytics in a real-time, flexible, interactive way.

For example, a business user in the airline industry can ask (in natural language) Adatao’s system to predict future airline delay ratios by quickly exploring 20 years of arrival/departure data (124 million rows of data) to break down past delays by week, month, and cause. In the same way Google Docs allows teams all over the world collaborate, Adatao allows data scientists and business users to collaborate on massive datasets, see the same views and together produce a visual model in just three seconds.

The Adatao software would not be possible, if not for the incredible team behind the project. I first met Christopher Nguyen, founder and CEO, at a breakfast meeting in Los Altos and was blown away by his humble personality. I knew at that moment, I wanted to find a way to partner with him. Here’s a guy who grew up in Vietnam and came to the US with a desire to make a difference. Since then, Christopher has started several successful companies, was engineering director of Google Apps and earned a PhD from Stanford and a BS from UC Berkeley, and is a recipient of the prestigious “Google Founders Award”.

He’s assembled a crack technical team of engineers and PhDs in parallel systems and machine learning. They all want to change the world and solve the most pressing data and information issues of our generation.

I am honored to be joining the board and look forward to partnering with this incredibly talented team. Adatao’s approach, team, and spirit of innovation will usher in a new generation of real-time, information intelligence that we believe will be the future of big data.