Explore the Latest Advancements in AWS Storage Solutions

Semi-structured Formats with Snowflake Support

The number of sources that produce semi-structured data has increased exponentially in recent years. The arrival of Snowflake Data Cloud has made it effortless to process complex datasets. Snowflake supports storing and processing semi-structured data. It supports semi-structured formats enlisted below:

• • JSON
• • Avro
• • ORC
• • Parquet
• • XML

This article is about the two features in Snowflake that come in handy for Data Engineers while handling semi-structured data.

• • Variant
• • Flatten

Variant

A variant is a datatype which can hold semi-structured data in a single field. Snowflake stores semi-structured data in the column format when semi-structured data is loaded into a VARIANT column. A single row can contain other underlying data. Let’s demonstrate with the help of examples:

The stored data can be easily retrieved and structured from the above table by a simple query as below:
select
v:time::timestamp as observation_time,
v:city.id::int as city_id,
v:city.name::string as city_name,
v:city.country::string as country,
v:city.coord.lat::float as city_lat,
v:city.coord.lon::float as city_lon,
v:clouds.all::int as clouds,
(v:main.temp::float)-273.15 as temp_avg,
(v:main.temp_min::float)-273.15 as temp_min,
(v:main.temp_max::float)-273.15 as temp_max,
v:weather[0].main::string as weather,
v:weather[0].description::string as weather_desc,
v:weather[0].icon::string as weather_icon,
v:wind.deg::float as wind_dir,
v:wind.speed::float as wind_speed
from json_weather_data
Note: ‘v’ is the Field (Column) name, and the JSON tags are arranged alongside to retrieve datasets.

Flatten

Compound values are flattened into multiple rows with the use of the Snowflake Flatten Command. A tool called Snowflake FLATTEN is used to transform semi-structured data into a relational structure. Relatively complex JSON structures such as nested JSON structures can be structured with FLATTEN function in Snowflake. Here is an example of the same: The above JSON file stores a single set of conversations between two texters in a single row. If the requirement is to get every individual message as a separate row a FLATTEN function can be used to produce the following result:

Here is the query to generate the above table from semi-structured data:

(SELECT
b.value:message_date::TIMESTAMP AS Time_Of_Message,
b.value:conversation_number::STRING AS Conversation_num,
b.value:message:message_text:msg_txt::STRING AS Message,
ROW_NUMBER() OVER ( PARTITION BY b.value:dialogId::STRING ORDER BY
b.value:message_date::TIMESTAMP ASC) as Message_order
FROM customer_messages t,LATERAL FLATTEN(input => t.v) b);

Summing Up

The ability of Snowflake’s assistance is undeniable. It aids you to make better decisions and offers a better overall experience when attempting to get the most out of your data, through features like Flatten and Variant. If you want to be a part of Snowflake’s incredible platform, let us help you. Beinex’s partnership with Snowflake enables us to offer you advanced features like automated tuning and elastic computing, along with analytics modernization services, to help your organisation realise exponential Return on Investment.

As a frequent user of Tableau Desktop and Server, check out my top picks from the list. • Improved dynamic parameter • Multiple marks layer support for maps • Block comments in calculations • Web authoring enhancements • Offline map support for Tableau Server • Tableau Server Management (TSM) improvements • Multiple key activation on Tableau Server prior to TSM initialization • Analytics Extension for Tableau Online • Merge duplicate external assets • Tableau Prep Builder in the browser Tableau Desktop Improved Dynamic Parameter The dynamic parameter was one of the features which I used extensively starting from 2020.1. The feature allows us to populate parameter values based on a specified field whenever the workbook is opened. We will be able to select only the field with matching data types. The feature update announced in Tableau 2020.4 gives me more privilege to customize the Dynamic Parameter. I can now set a parameter’s range to a field’s min and max value upon opening the workbook, or when the field’s data source is refreshed. So, whenever I open the workbook or the extract is refreshed, Tableau displays the unique values within the range specified, giving more control over the parameter values displayed. Multiple Marks Layer Support for Maps This is an exceptional feature to bring multiple spatial layers and context together to better understand and analyze geospatial data and map views. I will be able to include multiple marks layers from a data source to map visualizations and enhance the geospatial analysis. I can present more context in a single map view and perform further analysis with this feature. Block Comments in Calculations Block comments is a simple yet one of the most useful features for me, which Tableau has announced in 2020.4. I often used to add comments in complex calculations for easy understanding in future references. Earlier, only single-line comments were possible, limiting the description I could add. But from now on, I can add comments of any length to calculation windows with block comments by simply starting the comment with /* and ending with */. What makes it unique is that the new multi-line block comment feature is consistent with other popular programming languages. This feature is an example of Tableau’s on-going effort to provide its customers with an intuitive user experience. Tableau Server Web Authoring Enhancements Starting from Tableau 2020.4, it is possible to author dashboards from the browser just like how we design it in Tableau Desktop. With this web authoring enhancements, I can include Highlight actions, Format Mark Labels, apply filters to worksheets, create fixed sets, and even create extract in my workbooks from the browser itself. I no longer have to make changes from Tableau Desktop and publish it to the server. I can directly do it from the browser itself. Offline Map Support for Tableau Server Rendering dashboards with maps is now faster compared to previous versions. Now, I can create maps using the offline map style in web authoring, ensuring the performance of map views in Tableau Server. Offline map support is a great deal for organizations with strict internet access restrictions, assuring that map view access to all its users. Tableau Server Management (TSM) Improvements Tableau Server Administration activities like installation, upgrade and backup are now easy like never before. I can retry installation or upgrade from the last checkpoint in case of an unexpected issue or error during installation or upgrade, saving my efforts to obliterate tableau server. Backups can be performed twice as fast as previous versions, and I can monitor the progress with the new progress bar giving visibility into what step the backup is on and how much time is remaining. Backups can also be scheduled using TSM command starting from 2020.4 and that is awesome. I no longer need to prepare batch script and depend on the windows task scheduler to schedule the backups on regular intervals; instead, I can schedule it with just a single command. Multiple Key Activation on Tableau Server Prior to TSM Initialization During Tableau Server installation, it is now possible to activate multiple license keys prior to TSM initialization. I will be able to save a lot of time by eliminating the need to restart after the installation is completed to activate multiple licenses and experience a smoother installation. Analytics Extension for Tableau Online The power of the Analytics extension is now unlocked in Tableau Online too. The feature was already available in Tableau Server, and it helped us dynamically perform advanced analysis with models and functions in R, Python and other platforms. Analytics extension in Tableau Online significantly enhances the scope of using Advanced Analytics by the common users. Merge Duplicate External Assets Earlier, the Database or Table with similar names used to appear as multiple assets within Tableau Catalog. But, the new feature helps me to merge those multiple assets into a single one. I can manage assets easily and keep an organized view of External assets by merging the common ones. Tableau Prep Tableau Prep Builder in the Browser Prepare the data for visualizations from anywhere using a browser! Starting from 2020.4, Tableau is bringing the data prep process into one integrated platform on the web. Now I can easily prepare and manage prep flows from anywhere using a browser. Conclusion Tableau is progressively evolving as a single platform for data preparations, visualization, and collaboration with every update and version release. Author : Firdous Maqbool Images Courtesy : Tableau

The cloud-based approach delivers:

1. Better performance
2. More flexibility
3. Enhanced cost savings
4. Improved security
5. Facilitates excellent teamwork opportunities

Specifically, Tableau on AWS lets you process data more quickly and scale up or down resources as needed. Plus, you can access a range of AWS services to optimise your Tableau setup. AWS also provides tools to help you save money and a secure environment to protect against cyber threats and data breaches. Ultimately, Tableau on AWS enables teams to collaborate more efficiently, taking their data analysis and business intelligence to the next level.

There are several other benefits to using Tableau on AWS beyond scalability, cost, and security. Here are some additional insights:

1. Faster Deployment: With Tableau on AWS, you can deploy new instances of Tableau in minutes rather than days or weeks as you would with on-premises infrastructure. This is because AWS has pre-configured templates for Tableau that make it easy to spin up new instances quickly. 2. Better Performance: Tableau on AWS is designed to exploit AWS's high-performance infrastructure. Tableau runs faster and more efficiently on AWS than on traditional on-premises infrastructure. 3. Integration with Other AWS Services: Tableau on AWS integrates seamlessly with other AWS services, such as Amazon S3 for data storage, Amazon Redshift for data warehousing, and Amazon EMR for big data processing. This makes building a complete analytics solution easier by using Tableau and other AWS services. 4. Improved Disaster Recovery: With Tableau on AWS, disaster recovery is built. AWS provides automated backup and recovery services to quickly recover your Tableau environment and data if there is a disaster or outage. 5. Global Reach: AWS has data centres worldwide, meaning you can deploy Tableau in the region closest to your users for better performance. This is especially important for organisations with a global presence.

Overall, Tableau on AWS offers several advantages over on-premises infrastructure. By leveraging AWS's scalability, cost-effectiveness, and security, organisations can run Tableau more efficiently and with better performance. Additionally, AWS's integration with other services and global reach make it an attractive option for organisations looking to build a comprehensive analytics solution.

Tableau Server on AWS deployment options

The following list outlines the available options for deploying Tableau Server on AWS:
1. Self-Deployment on EC2 Instance: This option involves users provisioning and configuring an EC2 instance and deploying Tableau Server. This provides the most significant control over the deployment process and can be customised to specific needs. However, it also requires more expertise and effort from the user.
2. Quick Start Deployment: The Tableau Server on AWS Quick Start provides an automated deployment process using AWS CloudFormation templates. This simplifies the deployment process and ensures that best practices are followed. However, it may be less customisable than self-deployment.
3. AWS Marketplace Deployment: Tableau Server is also available on the AWS Marketplace with pre-built AWS CloudFormation templates. This provides a quick and easy way to deploy Tableau Server, with different pricing and instance options public. However, users may have less control over the deployment process than over self-deployment.

Users should evaluate their specific needs and expertise when selecting a deployment option. Self-deployment provides the most significant control and customisation, while Quick Start and AWS Marketplace deployment offer simplified and quick deployment options.

Our Top Cloud Engineering Services Include:

• Cloud Consulting • Cloud Security • Cloud Migration & Modernization and more Feel Free to Schedule a Call: https://beinex.com/cloud-engineering

Read More

Beinex Among Top BI Consulting Firms in the Middle East Beinex Ranked as Top Data Science Consulting Firms in the Middle East

About Beinex Cloud Services

By partnering with Beinex, a leading cloud engineering and enablement partner in the Middle East, you can leverage our expertise to optimize your cloud infrastructure, enhance performance, and ensure security and compliance. Our technical prowess allows us to design and implement tailored cloud solutions that align with your business needs, enabling you to maximize the benefits of cloud technology and drive innovation.

Partnering with industry-leading cloud technology providers like Google Cloud Platform, AWS, and Microsoft Azure to build scalable, secure, cost-effective solutions and deliver transformative cloud technology solutions. Beinex has also forged strong partnerships with Snowflake, Tableau, Alteryx, Alation, and Databricks. These collaborations enable us to offer comprehensive solutions that leverage the best-in-class capabilities of these platforms. We can provide our clients with cutting-edge technologies and expert guidance by working closely with our partners.

About Consultancy-me

Consultancy-me.com is a premier platform for those seeking to remain informed about the advisory and consulting industry. The website provides a comprehensive array of the latest news and trends across more than 60 industries and functional areas within the consultancy sector. Consultancy-me.com collaborates globally with over 150 prominent consulting firms and organizations.