Azure Data Explorer

Join Alex, Tory, and Rudy as they take on Azure Search, Data Explorer and Big Data.

Authors: Alex Gebreamlak, Tory Waterman, Rudy Sandoval

Azure Search, Azure Data Explorer and Big Data – Part 1 of 3 | Azure Search, Azure Data Explorer and Big Data – Part 2 of 3 | Azure Search, Azure Data Explorer and Big Data – Part 3 of 3

User Experience

The front end was built on Angular 8 with a reactive architecture using RxJS. The backend support was provided by DotNet Core 3.1 with SPA enabled. This allowed for close coupling of front-end components and backend controllers. Azure AD was also leveraged for authentication. 

When displaying large datasets, the user experience should be as simple as possible but not simpler. Since we were supporting multiple data structures; we felt it would benefit the user to have clear distinctions from one result set to the next. Creating a federated search result with clear distinctions seem to provide the best experience. 

UI Layout

Data visualization could give the user information about your data at a glance. With a bubble chart, your users could see which dataset contains the most content, views, or completed task.

Azure Data Explorer
The bubble chart
one example, there are a variety of options a simple bar chart can also be effective. Displaying the data from a different perspective gives the user more information without showing more data. 

After some research, Azure Data Explorer became a great candidate for handling big data problems. Azure Data Explorer is designed to handle petabytes of data. 

“Embed Azure Data Explorer in SaaS applications to ingest and analyze petabytes of data in real-time. Developers are using this data to monitor service and improve application performance, while business users are discovering user trends, creating personalized experiences, and developing new revenue streams.”  Microsoft

One of the biggest issues with using Data Explorer is that it uses Kusto. It is a query language started in 2014 as an internal Microsoft project to address Azure services’ needs for fast and scalable log and telemetry analytics.  

With that in mind, the plan was to create a wrapper for taking simple search queries and converting them to Kusto. This allows the user to take advantage of Data Explorer without learning a new query language. 

If you are considering a search solution and your dataset is large, Terabytes or even Petabytes, then Azure Data Explorer deserves a look. Use cases include IoT devices that generate billions of sensor readings. Analyzing this data typically requires multiple technologies, which slows the process, complicates maintenance, and has the potential to be unreliable. Businesses that rely on large volumes of log data to spot trends, patterns, or anomalies in near-real-time. Data Explorer offers two types of instance families depending on your workload needs with compute-optimized and storage optimized solutions. 

Azure Cognitive Search is great when you have data or documents that you need to run through an AI model. For example, if you have reviews, comments, or other text it can be run through a sentiment analysis model. This can allow you to search for negative reviews. Similar to this, if you have scanned documents they can be run through an OCR (Optical Character Recognition) model and can be made searchable. Cost is based on Search Units that can be scaled up or down to increase query performance or ingestion.