How AI Decision-Making is Improving Enterprise Outcomes
Blog
Share 
Infographics | Cloud Storage and On-Premise Storage: The Tug of War that Never Ends, Ends Now
29 April 2022 
Sanchit Katiyar Amidst cloud storage's ripening, corporate IT departments continue to weigh the pros and cons of on-premise storage vs cloud storage. Before making the right choice for your company, it is always better to analyse the differences between on-premises and cloud-based services and infrastructure.
For companies juggling a massive amount of data, cloud platforms are a boon. Many have adopted cloud storage for flexibility, ease of access, fast scalability, and cut short expenditures. Cloud deployments that are well-architected and managed offer significantly greater infrastructure flexibility and real cost-effectiveness. According to Gartner's prediction, by 2025, more than 95 percent of new digital workloads will be deployed on cloud-native platforms, up from 30 percent in 2021.
Related Articles
Best AWS Services & Practices Every Data Engineer Should Master in 2025: Everything You Need in Your AWS Toolkit
1. Start with Storage Using Amazon S3
Amazon S3 is a secure and reliable storage solution when you are dealing with massive datasets. It's highly scalable, extremely durable, and serves as a foundation for most data workflows. You can depend on it from initial data landing zones to backup archives.
2. Spin Up Power with Amazon EC2
When you need raw computing power for heavy-duty tasks, such as batch processing or running data pipelines, EC2 gives you the flexibility to choose instance types suitable for your workloads. You're in control of the compute environment, which is key for tuning performance.
3. Simplify ETL with AWS Glue
Managing extract-transform-load operations can be messy. AWS Glue resolves this with automated data discovery, code generation, and job orchestration. AWS Glue can support you if you're managing multi-source ingestion and need to clean and prepare your data for use.
4. Query at Speed with Amazon Redshift
Redshift offers the easiest and quickest way to run complex queries against large volumes of structured data. It's perfect for powering dashboards, reports, and business intelligence tools without the drag of traditional databases.
5. Tackle Big Data with Amazon EMR
If your workloads involve distributed computing using Apache Spark or Hadoop, EMR helps you deploy and manage those clusters in a fraction of the time. It is ideal for advanced data transformations and machine learning (ML) workloads, as it integrates easily with other AWS services.
6. Event-Driven Logic with AWS Lambda
Forget provisioning servers to process a few files. Lambda allows you to write lightweight, trigger-based code that responds to data events. It is an efficient serverless solution for processing files as they arrive or triggering downstream processes.
7. Streamline Real-Time Data with Amazon Kinesis
Modern data doesn't always arrive in neat batches; it streams in constantly. Kinesis helps you manage this chaos by capturing, processing, and analyzing real-time data. You can utilize it for use cases such as log monitoring, clickstream analysis, and sensor data processing.
8. Store Fast & Flexible Data with DynamoDB
DynamoDB is a fully managed, serverless database ideal for workloads where speed and uptime are paramount. It provides a NoSQL solution that works best in situations where low latency is essential, such as recommendation engines or personalized content delivery.
9. Keep Your Metadata in Check: Glue Data Catalog
The Glue Data Catalog can be considered as a metadata hub that consolidates information regarding datasets, schemas, and transformations for you. It improves discoverability and governance—two things no engineer should overlook.
10. Coordinate Workflows with AWS Step Functions
As you know, data workflows can span multiple tools, services, and dependencies. AWS Step Functions help you string those steps together into one cohesive flow, complete with retries and error handling. It's a visual way to orchestrate and manage complex processes with clarity and ease.
Best Practices for Using AWS Tools as a Data Engineer
AWS tools are powerful, but knowing what to use isn’t enough; how you use them is what drives real impact. That’s where the best practices for using AWS services come in:
• Scalability: Use services that grow with your data. Enable auto-scaling in EC2, EMR, and Lambda to handle variable workloads.
• Automation: Set up Glue jobs, Lambda triggers, and Step Functions to run tasks without manual effort.
• Security: Encrypt your data (both at rest and in transit) and adhere to least-privilege access with IAM roles.
• Cost Monitoring: Use spot instances, archive old data in S3 Glacier, and monitor costs with AWS Budgets.
• Smart Workflows: Break pipelines into smaller, reusable steps. Use Step Functions for clear orchestration.
• Track & Monitor Everything: Use CloudWatch and CloudTrail to keep an eye on performance, errors, and user actions.
• Organize Metadata: Keep your Glue Data Catalog updated and use clear naming so your data is easy to find and understand.
• Test Before You Trust: Validate your data and test your pipelines with sample loads before pushing to production.
• Document as You Go: You can easily maintain notes on your workflows, data sources, and transformations for smoother teamwork.
Wrapping Up: Why These Services Matter
Tools that enable speed, flexibility, and automation are not just desirable; they're essential. AWS offers a comprehensive toolkit that covers all stages of the data lifecycle. By staying up to date with these services, you not only improve your performance at work but also position yourself to take the lead in a data-driven, cloud-first future.
For data engineers seeking to excel in their roles, it is beneficial to become proficient in at least 10 AWS services. By serving as the foundation for scalable and effective data pipelines, these services help businesses transform unstructured data into actionable insights. Data engineers can significantly contribute to fostering innovation and informed decision-making within their companies by leveraging the potential of Amazon Web Services.
How EHSS reported a 40% increase in productivity after implementation of AWS
About Employee Health & Safety System (EHSS)
The Employee Health & Safety System (EHSS) is a comprehensive covid test and vaccination tracking platform developed by Beinex, designed to empower organizations to track and monitor covid related details of their own employees, external entities, vendors etc. vising their premises and derive meaningful insights from the data using a powerful analytics engine. This platform was developed to assist our client Department of Health, Abu Dhabi (DoH) to provide them a complete overview of covid 19 related status within the UAE. EHSS leverages cutting edge technology and the power of AI to remove some of the ardent pain points such as risk scoring of employees based on physiology, at the same time providing an intuitive user access control to manually add and edit employee records. EHSS helped DoH to visualize covid relation information by combining four independent processes in silo, onto one single integrated platform – employee health tracking, input and data capture, reporting & analytics and collaboration & control.Challenge
The IT infrastructure of DoH where EHSS platform was hosted was in an on-premise datacenter which was designed to be scalable and robust with multi node physical clusters including the server, storage and network components. However, most of the physical hardware was quite old and not equipped with the latest generation of physical servers.
Frequent hardware crashes and portal downtime kept troubling the availability of the EHSS application. Assigning a touch hand support person to power on the hardware that was down seemed quite impossible due the restrictions during covid period. Hence, DoH wanted to look for another viable solution.
Though the hardware setup at DoH was well equipped to meet the occasional spikes in the traffic, it was observed that over a course of 6-month time, most of the IT infra was underutilized than predicted. It was realized that spending huge amount of money on an old hardware plus software maintenance, license costs, internet bandwidth, datacenter cooling and maintenance, touch support personnel and electricity costs – were keeping the business operations challenging.
There was an attempt by DoH to select a cost-effective solution that can host EHSS application servers, web servers and archival data. This way IT infra can be re-provisioned to host sensitive data on-premise and the rest on the cloud, thereby reducing the overall physical hardware costs spent on a yearly basis.
Why AWS
Auraa’s backend application servers, web servers, database servers and archival data were decided to be migrated to AWS.
DoH decided to migrate EHSS’s backend application servers, web servers, database servers and archival data to AWS.
The EHSS application’s AWS architecture includes Amazon Elastic Compute Cloud (Amazon EC2), that provides complete control of its computing resources, updates to tables in Amazon Relational Database Service (Amazon RDS) and AWS Elastic Load Balancer was used to distribute the traffic to the underlying EC2 instances based on the load.
Exploration of Amazon ElastiCache to retrieve data from fast, managed in-memory caches is in progress by DoH. This technology will be used to build distributed data collection systems and handle real-time statistics and metadata associated with EHSS application.
Benefits
DoH uses AWS services now to provision infrastructure and deploy the EHSS platform to other departments within it. In addition, the EHSS resources that are no longer required to be run all the time are made to auto shutdown thus saving cost. DoH reports an increase of 40% productivity after implementation of AWS for the EHSS platform.
The implementation of EHSS on AWS made DoH confident in the security of its data, and its accreditation team is enthusiastic about the monitoring and auditing capabilities provided by AWS tools. With the implementation of IAM roles, DoH IT team was able to isolate systems and tightly control user accesses. These capabilities were harder to achieve within the existing infra but were available out of the box with AWS
By adopting AWS to host the EHSS platform, DoH has been able to innovate and experiment to a degree previously impossible. For example, DoH compared the performance and cost-effectiveness of three different cloud solutions. Without moving to the AWS, the costs associated with running an outdated on-premise hardware would have creeped up and the alternative way of upgrading the existing on-prem infrastructure to the latest hardware models and then hosting the EHSS application on top of it would have taken months.
Coming Soon: Tableau 2022.1 – What Does It Offer New
- List the unused data sources: Data sources imported in a workbook but not used are highlighted by the new feature. The developer can remove these data sources from the data source list to make the workbook faster.
- List of unused fields or data columns: Just like the data sources, workbook optimizer also highlights the data columns not used across the workbook. Removing these at the data source level can help improve the overall performance of the workbook.
- List of sheets not used in the dashboard: It is a common practise that developers tend to create sheets not used in the final dashboard. This creates unnecessary clutter and makes the dashboard slower. The workbook optimiser feature gives a list of such sheets which the user can delete to optimise the performance of the workbook further.
- Highlights lengthy calculations: The feature provides a list of calculations which are too complex and in turn reduce the performance of the dashboard. Simplifying a few of these can improve performance to a great extent.
Ask Data Phrase Builder: This feature is available on Tableau Server and Tableau Online
Add field would look like as shown in the below screenshot:
Customize View Data: This feature is available on Tableau Server, Desktop and Tableau Online
This feature enables to reshape the tabular data behind your visualisation in the View data interface. One can create new columns, remove columns from the default view, change the order and sort the data using this feature. This reshaped data can also be exported as csv file to be shared with the team.
Change the root table: This feature is available on Tableau Server, Desktop and Tableau Online
Managing multiple data tables becomes easier and flexible with this new feature. One can swap any table to be the root table with a single click. This allows one to change the layout of the table quickly, reshape the data with a different root table and delete a specific table without deleting child nodes. For e.g., let us assume a user had to create a data source for an analysis using 3 tables namely ‘customers’, ‘orders’ and ‘returns’. The user creates the data model such that ‘customers’ is the root table followed by ‘orders’ and ‘returns’. But after performing some analysis the user realises that ‘orders’ should be the main root table. In such cases the user would have to re-create the data again from scratch but with the new ‘Swap with root table’ feature user can do the changes with a few clicks.
Parameter Enhancements in Tableau Prep
In version 2022.1 Tableau Prep is adding even more places where one can use parameters in the flow as well as user enhancements. Now, one can:
- Get a list of all the parameters in one place rather than finding them in the flow. One can delete these parameters directly from the parameter window rather than to find it in the flow first.
- Include parameter names in exported output files.
- Include parameters in SQL scripts that you run before or after writing the flow output to a database. Include parameters in worksheet names when writing the flow output to Microsoft Excel.
- 1. Improvements in Esri Data Connector
- 2. Addition of new Accelerators
- 3. Added connectors to connect to more data sources
TABLEAU 2019.2 RELEASE
Tableau 2019.2 Release – A Deeper Dive
Tableau has released the newest version of the platform that enhances the way people visualize and interact with data. In this article we’ll go through some of the exciting new changes included in Tableau 2019.2 and how business will benefit from these newly enhanced and released features.
The major release of Tableau has on-boarded some impressive features and functionality that will have big influence in data visualization and business productivity. In this post, we will highlight some significant and exciting features that are a part of this release.
1. Upscale your Viz with Vector Maps:
Vector Maps enhances your mapping experience by rendering a crisp output as you pan, zoom in, and zoom out to explore your geospatial data. This new feature also leverages the census data from American Community Survey (ACS) to incorporate demographics data into your dashboard.
Highlights:
Enhanced mapping interaction with sharp rendering
Improved default maps styles: Dark, Normal, and Light
New map styles: Street, Satellite, and Outdoor
New background mapping styles: subway and train stations, building footprints, terrain, and water labels
Demographics data from American Community Survey (ACS)
Usability:
Above recording is a view of the new map styles that is a part of the Tableau 2019.2 release. Here you can see how simple it is to include and switch between the newly added map layers. The maps have become crisper while zooming and panning.
2. Enable Interactivity with Parameter Actions
Improved parameter actions is another powerful feature that powers up interactivity in your dashboards and help your viewers gain deeper insights into the trends. It unlocks the ability of a viewer to visually change a parameter’s value thereby offering you endless possibilities to create a truly interactive dashboard.
Highlights:
Interactive relationships between data, dashboard, objects, other workbook sheets and web
Enhanced interactivity by creating marks on a Viz
Improved user experience by transferring control to the end user
Allows users to dynamically change SQL queries, drive reference lines, calculations and filters
Usability:
With this feature the end users can dynamically view the parameter values. In the above example, we have created a parameter ‘order date’ and placed Day of order date and sales into the view and we have added reference line for sales. When applied in a parameter action, the values get changed dynamically.
3. Left Nav, Favorites, and Recent for Enhanced Navigation
Navigate seamlessly and find the content you are searching with the improved navigation. The new navigation features will intensify the already powerful content browsing experience. You can easily find your favorite and recent content including projects and prep flow at the top.
Highlights:
Improved access to key pages and content
New quick access navigation bar with personalized content relevancy
Welcome screen with navigation walk-through for new users
Completely re-designed homepage
Usability:
4. Talk to your data with AskData
AskData is a fairly new module and growing quickly in popularity with the 2019.1 release. Tableau is continuing to expand on this by adding streamlined and innovative capabilities to AskData. You now have the option to add calculations on the fly without having to build them into the data source. You can also ask more sophisticated questions and get answers with the enhanced AskData.
Highlights:
Improved interface to allow more conversational interactions
Enhanced language processing to edit statements from an existing question
Features to execute calculations on the fly
Usability:
In the above example, you can experience the power of the natural language processing engine built into the latest version of ask data. When we ask ‘How are my sales doing by numbers?’, the system automatically understands our query and displays the sum of sales by segment. We can further segment it by year by asking further questions or adjust the values according to our requirements.
These are only a few of the many exciting updates Tableau has incorporated to continue pushing the boundaries of data analytics and prepare for the future market changes. A complete list of updates and features could be found in Tableau’s release notes: https://www.tableau.com/products/all-features
Beinex is a digital transformation organization en-rooted with ideas, innovation and unparalleled customer service. Our mission is to transform the way individuals and the organizations work with the data through innovation and experience.
If you are interested in learning more about the latest Tableau 2019.2 features and updates, please contact us at training@beinex.com/ info@beinex.com and we would be happy to schedule a Tableau demo or training for you and your company.
Data Blending for Spatial Analytics in Just 6 Steps Using Alteryx
What is Spatial Analysis?
Spatial analysis is the art and science of extracting insights from data that has a geographic component. Think of it as giving your data a physical address! Traditionally, this involved complex Geographic Information Systems (GIS) software. But today, spatial analysis is more accessible than ever, thanks to data science and machine learning.
Pinpointing locations on a map is just the first step. Spatial analytics goes far beyond that, offering a powerful lens to understand how relative location impacts your business. It allows you to see the bigger picture: how customers, stores, services, and other factors interact with each other geographically.
This magic happens by blending spatial data (think zip codes, store addresses) with your existing data sets (sales figures, customer demographics). By analyzing these combined datasets, you gain a wealth of insights that can transform your decision-making process.
What Spatial Analytics Can Do for You
Here's how spatial analytics can help you understand and optimize key areas:
- • Customer Behavior and Inventory: Analyze nearby consumer buying habits for specific products and services. This allows you to customize inventory and service experiences at each location based on local demand.
- • Strategic Location Planning: Optimize your location strategyby determining how the proximity of competitors or existing locations impacts new site expansion. You can also understand how far customers are typically willing to travel for your product or service.
- • Improved Customer Experiences: Ensure service availabilityand minimize service gaps by strategically locating key hubs within an appropriate distance from each other. This translates to a smoother and more efficient experience for your customers.
- • Targeted Marketing: Drive efficiencies in your marketing programs by customizing your offerings to match demographic purchasing preferences in specific locations. This targeted approach allows you to reach the right audience with the right message.
- • Import and Unify: Easily bring in various datasets, regardless of format.
- • Effortless Geocoding: Transform addresses and other location data into usable geographic coordinates with a few clicks.
- • Spatial Blending: Combine your location data seamlessly with traditional datasets for a holistic view.
- • Advanced Analytics Made Simple: Perform complex spatial analyses without needing specialized coding skills.
- • Data Enrichment: Boost your insights by adding demographic, firmographic, or industry-specific data to your spatial datasets.
- • Visualize and Explore: Discover hidden patterns and relationships through interactive maps and visualizations.
- • Gather Data: Gather all the data sets you need for your analysis, from customer information to market demographics.
- • Translate Your Addresses: Use Alteryx's geocoding tools to transform addresses and other location data into usable geographic coordinates.
- • Define Your Trade Zone: Create a virtual boundary to analyze specific locations based on radius
- • Blend Datasets Together: Seamlessly combine your spatial data with traditional datasets to create a comprehensive picture of customer-location relationships.
- • Use Advanced Spatial Analytics for Additional Insights: Perform complex spatial analyses within Alteryx's user-friendly interface, unlocking hidden insights without needing specialized coding skills.
- • Visualize and Share Your Findings: Prepare your data for reports and interactive visualizations that effectively communicate your insights. Alternatively, export the data for further analysis or integration with downstream processes.
How Alteryx Enables Data Blending for Spatial Analytics
Forget complex GIS! Alteryx's no-code tools make spatial analysis a breeze, unlocking location intelligence for all data users. Optimize resources, plan assets, manage logistics, and more - all in a user-friendly platform.
Alteryx offers an intuitive workflow that streamlines the entire process:
A 6-Step Recipe for Blending Spatial Data in Alteryx
Alteryx makes blending spatial data with your existing information a breeze. Here's a step-by-step guide to get you started:
Find a detailed 6-step guide for blending spatial data using Alteryx:
1. Gather Data
Alteryx's Input tool lets you grab data from anywhere – spreadsheets, databases, even social media! Just connect to your desired sources, and Alteryx will get your data ready for spatial exploration.
2. Turn Addresses into Locations:
The Street Geocode tool in Alteryx quickly transforms your standard addresses (like customer locations or branch sites) into geographic coordinates (latitude and longitude). This "spatializes" your data, adding a new data point for each location.
In this example, we'll use it to geocode both customer and site data.
3. Define Your Trade Zone:
The Trade Area tool lets you see what's happening within a specific area around each location. For example, you can create a 10-minute drive time polygon. This "draws" a zone around each location, encompassing all areas reachable within a 10-minute drive using the road network.
4. Blend Datasets Together:
The Spatial Match tool lets you see how different sets of locations relate to each other. For instance, you can use it to find out how many customers live within (or outside) the 10-minute drive time zone you created for each location. It essentially compares your customer data points (spatial points) with the trade area polygons (spatial objects) to identify matches based on spatial relationships (like "contains" or "intersects").
5. Use Advanced Spatial Analytics for Additional Insights
Alteryx offers a range of additional tools for advanced spatial analysis, making it accessible to users beyond data specialists. Additional tools include:
6. Visualise and Share Your Findings:
Alteryx doesn't just help you crunch data - it enables you to share your insights clearly.
Visualize Your Success: Overlay data on detailed maps or satellite imagery using advanced mapping tools.
Spread the Knowledge: Export your analysis in various formats like Excel, ESRI, or even Tableau and Qlik for seamless integration with other data workflows and presentations.
Beinex + Alteryx Offerings
As a Premier Alteryx partner, we have extensive experience and a proven track record of success. Our team is highly skilled in Alteryx solutions and can help you unlock the full potential of this powerful platform.
Contact us today to learn more about how Alteryx and our partnership can take your business to the next level.
Image Source: https://community.alteryx.com/pvsmt99345/attachments/pvsmt99345/general-discussions/2303/2/Spatial_Cookbook_Victa.pdf