Azure HDInsight — Technology Profile & Architecture

Core Information

#core_information

Name, description, vendor website, logo, technology type and tags.

Overview & Role

Managed open-source analytics clusters for Hadoop, Spark, Kafka and HBase.

Domain Classification & Tags

Analytics

Performance Profile

#performance_profile

Typical read and write throughput, payload size and processing latency.

Read Performance Profile

Throughput: Varies significantly by cluster type and workload. For Spark/Interactive Query: hundreds of MB/s to several GB/s per cluster. For Kafka: hundreds of MB/s to several GB/s per broker.

Payload Size: Varies by workload. For analytical queries (Spark/Hive): typically large datasets (MBs to GBs per query). For Kafka: messages from bytes to MBs. For HBase: rows/cells from bytes to KBs.

Processing Latency: Varies significantly by cluster type and workload. For Interactive Query (LLAP): sub-second to a few seconds for complex queries. For Spark: seconds to minutes for batch jobs, sub-second for streaming micro-batches. For Kafka: ~10ms for low-latency consumers. For HBase: single-digit to tens of milliseconds for point reads.

Write Performance Profile

Throughput: Varies significantly by cluster type and workload. For Spark/Hadoop: hundreds of MB/s to several GB/s per cluster for batch writes. For Kafka: hundreds of MB/s to several GB/s per broker. For HBase: thousands to tens of thousands of writes per second per region server.

Payload Size: Varies by workload. For analytical jobs (Spark/Hadoop): large datasets (MBs to GBs). For Kafka: messages from bytes to MBs. For HBase: rows/cells from bytes to KBs.

Processing Latency: Varies significantly by cluster type and workload. For Spark/Hadoop: seconds to minutes for batch jobs. For Kafka: ~10ms for producers with appropriate acknowledgments. For HBase: single-digit to tens of milliseconds for single writes.

Architecture Diagram

#architecture_diagram

Reference diagram of the technology's internal architecture.

Component Architecture & Topology

open_in_new Standalone Tab

Features

#features

Catalogued product capabilities and what each one does.

Empirical data for features is currently being compiled in the global catalog.

Typical Use Cases

#typical_use_cases

Scenarios the technology is commonly chosen for.

Empirical data for typical use cases is currently being compiled in the global catalog.

Known Customers

#known_customers

Publicly referenced organisations using the technology.

Empirical data for known customers is currently being compiled in the global catalog.

Known Integrations

#known_integrations

Other products and services it is documented to work with.

Empirical data for known integrations is currently being compiled in the global catalog.

Connectors

#connectors

Directional data connections to other technologies, with direction and maturity.

Empirical data for connectors is currently being compiled in the global catalog.

Reference Architectures

#reference_architectures

Published architectures where the technology is used or mentioned.

No published reference architecture blueprints currently link to Azure HDInsight.

Security Features

#security_features

Built-in security and access-control capabilities.

Empirical data for security features is currently being compiled in the global catalog.

Known Issues

#known_issues

Documented limitations, defects and operational pitfalls.

Empirical data for known issues is currently being compiled in the global catalog.

Guidelines

#guidelines

Recommended practices for adopting and operating the technology.

Empirical data for guidelines is currently being compiled in the global catalog.

Standards & Compliance

#standards_and_compliance

Standards, certifications and control requirements it maps to.

Empirical data for standards & compliance is currently being compiled in the global catalog.

Sources

#sources

Documentation and research references behind the recorded information.

Empirical data for sources is currently being compiled in the global catalog.

Expert Validation

#expert_validation

Whether domain experts reviewed and confirmed the content.

Empirical data for expert validation is currently being compiled in the global catalog.