BIG DATA

Amazon Web Services Inc. is reinventing the way data-intensive analytical workloads are handled by the Amazon OpenSearch Service with the launch of a new, purpose-built log analytics engine today.

The cloud computing giant says the new engine is able to reduce data storage costs by around 70% on average while doubling data ingestion throughput. That way, organizations can conduct analytical queries twice as fast as before, without adapting the underlying hardware configuration.

Amazon OpenSearch Service is a fully managed version of the open-source OpenSearch platform, which is designed to handle distributed search and analytics workloads. It’s used for tasks such as real-time application monitoring, log analytics, web search and security monitoring, as well as broad-based observability applications.

The service can ingest massive streams of data from sources including cloud infrastructure, applications and network devices, before being sent onward to populate interactive dashboards that track system health and performance and identify operational problems.

With the update, AWS is trying to get past a critical bottleneck for enterprise DevOps teams. In a blog post written by AWS solutions architects Jagadish Kumar and Michael Suphagat, and product manager Rohin Bhargava, the company explained that most organizations are seeing their log analytics volumes rise by around 30% to 40% each year.

That’s because they’re dealing with a growing deluge of data from artificial intelligence applications. As a result, many now have to choose between increasing their budgets and adding a secondary analytical database to keep up, or simply delete much of their log data without even analyzing it.

AWS’s solution is an entirely new, more optimized log analytics engine that moves Amazon OpenSearch Service away from its inflexible one-size-fits-all storage model. The new engine is accessible as a selectable mode within the Amazon OpenSearch Service domain, and allows customers to step up log data ingestion while maintaining their existing consoles, security architectures and network configurations, the authors said.

The engine does this because it introduces new approaches to both storage and data processing. On the storage side, it can now store information in the columnar Apache Parquet format, rather than use traditional inverted indexes. That means data availability can be optimized using techniques such as dictionary encoding and tight numeric packing, eliminating the per-document overhead that comes with JavaScript Object Notation storage formats.

As for data processing, the new engine introduces intelligent query routing that leverages the open-source Apache Calcite tool to parse through incoming queries and automatically send them to the most efficient sub-engine. So deeper analytical operations on columnar data can be handled by Apache DataFusion, with standard queries sent to Apache Lucene.

It also supports unified query execution, because full-text search predicates such as MATCH and MATCH_PHRASE now work natively within both Structured Query Language and Piped Processing Language queries. Teams can now filter broader aggregations and drill down into specific point-lookups with a single query, the authors explained.

Kumar, Suphagat and Bhargava shared a series of internal benchmark results illustrating how the new engine can speed up log analytics at massive scales. In their tests, the engineers analyzed 24.4 billion documents and 9.5 terabytes of raw JSON data, achieving an ingestion throughput of 1.78 million documents per second, which is roughly twice as much as what the standalone Apache Lucene can handle.

Moreover, it did this with reduced computational requirements. The benchmark also showed the engine’s ability to process time-filtered analytical queries over billions of log events in milliseconds, with a four-times improvement in overall price performance.

Perhaps the best news is that these performance gains are being made available to all AWS customers from today. The optimized engine can be accessed through Amazon OpenSearch Service in all AWS regions where OpenSearch Optimized Instances are available.

Users must select the new engine at domain creation time by choosing “observability” as the primary use case from within the AWS console. Pricing is based on the standard charges for instances and storage, with no premium.