إصدار تابلو 2019.2

1. Snowflake Iceberg Tables Now Generally Available

Snowflake Iceberg Tables have moved to general availability, offering full storage interoperability with the Apache Iceberg open table format. This feature facilitates easier governance and collaboration on Iceberg data stored externally, enhancing the flexibility of data lakehouses, data lakes, and data meshes. Over 300 customers have already adopted Iceberg in its public preview, highlighting its potential to broaden Snowflake's data footprint.

2. Advancements in Snowflake Cortex AI

Snowflake introduced several enhancements to Cortex AI, including:

• • Cortex Analyst: Built with Meta's Llama 3 and Mistral Large models, this tool allows businesses to build applications securely on top of Snowflake's analytical data.
• • Cortex Search: Leveraging Neeva's retrieval and ranking technology, it facilitates the development of apps against documents and text-based datasets.
• • Cortex Guard: Aiming to ensure model safety, it filters and flags harmful content, including violence and hate speech.
• • Document AI: This feature, powered by Snowflake Arctic-TILT multimodal LLM, will soon allow users to extract data from documents such as invoices and contracts.

3. Introduction of Polaris Catalog

The Polaris Catalog is a vendor-neutral, open catalog implementation for Apache Iceberg, providing cross-engine interoperability and greater flexibility. It will become open-sourced within 90 days, supporting a variety of engines, including Apache Flink, Apache Spark, and Trino.

4. Private Preview of Snowflake Horizon Updates

Snowflake launched a private preview of an internal model marketplace within Snowflake Horizon. This marketplace enables users to publish and curate models, applications, and data products for internal use ensuring controlled access and preventing unintended external sharing. Other upcoming features include AI model sharing and AI-powered object descriptions.

5. AI & ML Improvements

• • Snowflake AI & ML Studio: A no-code interactive interface for AI development, now in private preview.
• • Cortex Fine-Tuning: In public preview, allowing customization of pre-trained models for specialized tasks.
• • ML Lineage: In private preview, offering traceability across ML life cycles.
• • Feature Store: Now in public preview, for creating, managing, and serving ML features.

6. Snowflake Native Apps with Snowpark Container Services — Preview

Snowflake introduced the integration of the Native App Framework with Snowpark Container Services on AWS. This integration provides developers with configurable GPU and CPU instances for various applications, from computer vision to geospatial data analysis. Over 160 Snowflake Native Apps are now available in the marketplace.

7. Developer Tool Enhancements

Several updates aimed at developers include:

• • Snowflake Notebooks: Snowflake Notebooks is now in preview, offering an interactive, cell-based programming environment for Python and SQL within Snowsight. It enables exploratory data analysis, machine learning model development, and other data science tasks all in one place.
• • Snowpark pandas API: Allows the use of pandas syntax for AI and pipeline development within Snowflake. The Snowpark pandas API is now in preview, allowing you to run pandas code directly on Snowflake data. This API offers a pandas-native experience with Snowflake's scalability and security, handling larger datasets without rewriting pandas pipelines.
• • Database Change Management: A public preview feature for DevOps, including Git integration.
• • Python API and CLI: Soon to be generally available, facilitating CI/CD practices.

8. Expanded Cloud Footprint and Governance

Snowflake announced a new data boundary for the EU, ensuring regional data residency and compliance. Additionally, a Department of Defense (DoD) environment meeting IL4 security controls will be available, highlighting Snowflake's commitment to robust data governance and security.

9. Snowflake Trail

The new Trail set of observability capabilities was unveiled, providing developers with tools to monitor, troubleshoot, and optimize workflows. Trail integrates with platforms like Grafana, Metaplane, and Slack, adhering to OpenTelemetry standards.

10. Snowflake Cortex Fine-Tuning — Preview

Cortex Fine-Tuning, now in preview, lets users adapt pre-trained models for specialized tasks. This managed service fine-tunes popular large language models using your data within Snowflake, enhancing model performance for specific use cases.

The preview of Snowpark Native Apps with Snowpark Container Services enables running containerized services within Snowflake Native Apps. This feature supports provider IP protection, security, data sharing, monetization, and integration with compute resources.

11. Snowpark Python Local Testing Framework — General Availability

The Snowpark Python local testing framework is now generally available. This emulator allows you to test Python code locally with Snowpark Python DataFrames, facilitating development and CI pipeline integration without needing a Snowflake account connection.

12. Universal Search and Snowsight Updates

Universal Search, now generally available, allows users to search for content across Snowflake storage, external Iceberg storage, and third-party providers. Snowsight also received a dark mode feature, enhancing user experience in low-light conditions.

13. New Geospatial Functions in Preview

Four new functions for GEOGRAPHY objects are now available in preview:

• • H3_TRY_COVERAGE: A special version of H3_COVERAGE that returns NULL if an error occurs when attempting to return an array of IDs (INTEGER values) identifying the minimal set of H3 cells that completely cover a shape.
• • H3_TRY_COVERAGE_STRINGS: Similar to H3_TRY_COVERAGE but returns hexadecimal IDs (VARCHAR values).
• • H3_TRY_POLYGON_TO_CELLS: Returns an array of INTEGER values of the IDs of H3 cells with centroids contained by a Polygon, returning NULL if an error occurs.
• • H3_TRY_POLYGON_TO_CELLS_STRINGS: Similar to H3_TRY_POLYGON_TO_CELLS but returns VARCHAR values.
With innovations in AI, data governance, and developer tools, Snowflake continues to drive forward the capabilities of its platform, ensuring customers can leverage data more effectively and securely. The future looks promising as Snowflake expands its offerings and strengthens its ecosystem, providing powerful solutions.

Semi-structured Formats with Snowflake Support

The number of sources that produce semi-structured data has increased exponentially in recent years. The arrival of Snowflake Data Cloud has made it effortless to process complex datasets. Snowflake supports storing and processing semi-structured data. It supports semi-structured formats enlisted below:

• • JSON
• • Avro
• • ORC
• • Parquet
• • XML

This article is about the two features in Snowflake that come in handy for Data Engineers while handling semi-structured data.

• • Variant
• • Flatten

Variant

A variant is a datatype which can hold semi-structured data in a single field. Snowflake stores semi-structured data in the column format when semi-structured data is loaded into a VARIANT column. A single row can contain other underlying data. Let’s demonstrate with the help of examples:

The stored data can be easily retrieved and structured from the above table by a simple query as below:
select
v:time::timestamp as observation_time,
v:city.id::int as city_id,
v:city.name::string as city_name,
v:city.country::string as country,
v:city.coord.lat::float as city_lat,
v:city.coord.lon::float as city_lon,
v:clouds.all::int as clouds,
(v:main.temp::float)-273.15 as temp_avg,
(v:main.temp_min::float)-273.15 as temp_min,
(v:main.temp_max::float)-273.15 as temp_max,
v:weather[0].main::string as weather,
v:weather[0].description::string as weather_desc,
v:weather[0].icon::string as weather_icon,
v:wind.deg::float as wind_dir,
v:wind.speed::float as wind_speed
from json_weather_data
Note: ‘v’ is the Field (Column) name, and the JSON tags are arranged alongside to retrieve datasets.

Flatten

Compound values are flattened into multiple rows with the use of the Snowflake Flatten Command. A tool called Snowflake FLATTEN is used to transform semi-structured data into a relational structure. Relatively complex JSON structures such as nested JSON structures can be structured with FLATTEN function in Snowflake. Here is an example of the same: The above JSON file stores a single set of conversations between two texters in a single row. If the requirement is to get every individual message as a separate row a FLATTEN function can be used to produce the following result:

Here is the query to generate the above table from semi-structured data:

(SELECT
b.value:message_date::TIMESTAMP AS Time_Of_Message,
b.value:conversation_number::STRING AS Conversation_num,
b.value:message:message_text:msg_txt::STRING AS Message,
ROW_NUMBER() OVER ( PARTITION BY b.value:dialogId::STRING ORDER BY
b.value:message_date::TIMESTAMP ASC) as Message_order
FROM customer_messages t,LATERAL FLATTEN(input => t.v) b);

Summing Up

The ability of Snowflake’s assistance is undeniable. It aids you to make better decisions and offers a better overall experience when attempting to get the most out of your data, through features like Flatten and Variant. If you want to be a part of Snowflake’s incredible platform, let us help you. Beinex’s partnership with Snowflake enables us to offer you advanced features like automated tuning and elastic computing, along with analytics modernization services, to help your organisation realise exponential Return on Investment.

AWS Network Firewall Expansion

AWS Network Firewall, a managed firewall service, now expands its availability to four additional AWS Regions. This expansion ensures that organisations worldwide can benefit from the advanced network protection it provides. With AWS Network Firewall, users can effortlessly enforce network security rules and gain granular control over traffic flow. This enhanced capability fortifies the first line of defence, safeguarding against potential threats and attacks.

Enhanced Management with AWS Security Hub and AWS CloudFormation

AWS Security Hub announces a significant improvement in management capabilities by integrating with AWS CloudFormation. This powerful integration simplifies managing security and compliance resources across the AWS infrastructure. Now, organisations can automate the deployment and configuration of security standards using Infrastructure as Code (IaC) principles, ensuring consistent and secure cloud environments across their operations.

AWS Control Tower Integration

The general availability of AWS Control Tower's integration with Security Hub marks a crucial milestone in maintaining control and compliance over AWS accounts. AWS Control Tower streamlines setting up a well-architected and compliant multi-account environment. With Security Hub integration, administrators gain enhanced visibility and control over security findings, enabling enforcement of centralised governance and security best practices.

You can now activate more than 170 Security Hub detective controls, aligning them with corresponding control objectives from AWS Control Tower. Notably, AWS Control Tower can now detect when a control is disabled in Security Hub, indicating a 'Drifted' control state. This new drift detection capability simplifies the monitoring of control deployment status, enabling you to promptly manage the security posture of your AWS Control Tower environment by taking necessary actions.

Amazon Inspector Code Scans for AWS Lambda Function

Ensuring secure serverless applications are now more accessible with the general availability of Code Scans for AWS Lambda functions. Amazon Inspector Console, a robust security assessment service, now offers code-level security assessments for Lambda functions, identifying vulnerabilities and potential security risks. This capability empowers developers to take proactive measures to strengthen the security posture of their serverless applications.

Amazon Verified Permissions

With the introduction of Amazon Verified Permissions, AWS simplifies the permissions management process for cloud resources. Organisations can streamline security audits and compliance checks, saving time and effort. This capability provides an extra layer of confidence, ensuring that only authorised users can access and modify critical resources.

Utilise your current identity provider, responsible for managing users and groups, to effectively manage application permissions and control access. With this integrated authentication and authorisation solution, applications utilising Amazon Cognito now benefit from seamless policy validation based on attributes in Amazon Cognito while also being able to authorise requests using Amazon Cognito tokens.

AWS Security Hub Automation Rules

AWS Security Hub Automation Rules introduce a game-changing capability for proactive incident response. Users can now define automated actions responding to security events, enabling faster and more efficient incident resolution. This empowers organisations to respond swiftly to potential security threats and minimise the impact of security incidents.

AWS Global Partner Security Initiative

Security is a shared responsibility, and AWS takes a collaborative approach to empower its users and partners in fortifying their cloud security. The AWS Global Partner Security Initiative provides valuable insights, resources, and tools to help partners enhance their security offerings and better protect their customers' data. This initiative fosters a more robust security ecosystem and builds trust among AWS users worldwide.

AWS continues to demonstrate its commitment to cloud security with these latest capabilities. As organisations navigate an ever-changing threat landscape, the robustness of AWS security offerings ensures that cloud environments remain fortified and data remains secure. By leveraging these new features and capabilities, users can confidently embrace the cloud's potential without compromising security.

Beinex Offering

Beinex is an AWS consulting partner, and we empower customers to host their BI solutions and much more on the cloud. Our cloud migration experts bring in best-in-class stability and reliability by understanding your business strategy and working closely with you to deploy AWS infrastructure as a service.

Top Cybersecurity Trends

The cybersecurity measures are uncompromising, and leaders should be informed about the changing trends and update frequently. Here are some important cybersecurity trends that are going to help you:

1. The Rise of Cybersecurity Threats in the Automotive Industry

Sophisticated software in modern cars provides smooth communication and cutting-edge functions, including driver assistance systems, engine timing, and cruise control. However, cars are also vulnerable to potential hacking threats due to their reliance on automation and the internet. Strict cybersecurity safeguards are required, especially for self-driving or autonomous vehicles, as these dangers are predicted to increase with the growing use of automated vehicles.

2. Blockchain

Although blockchain technology is most commonly associated with cryptocurrency, it could also revolutionize cybersecurity. Blockchain can offer safe storage for private data by establishing a decentralized database. It is far more difficult for hackers to obtain illegal access because there is no central authority in charge of the data.

3. Quantum Computing

This technology processes data by utilizing the principles of quantum mechanics. Compared to conventional computers, it may potentially tackle complicated problems far more quickly. Despite its early stages, this technology has the potential to completely transform cybersecurity by enabling more secure encryption.

4. Leveraging Artificial Intelligence's Potential in Cybersecurity

Artificial intelligence (AI) has emerged as a vital element for enhancing cybersecurity in various industries. AI has enabled the creation of automated security systems that can perform tasks such as danger identification, facial detection, and natural language processing through machine learning algorithms. Fraudsters, however, also utilize this same technology to create sophisticated attacks designed to circumvent security measures. Despite these difficulties, threat detection systems powered by AI can react quickly to new threats, which is crucial assistance for cybersecurity experts.

5. Mobile Devices: A Common Cyberattack Target

Due to their widespread use, mobile devices have become easy targets for cybercriminals, as proved by the sharp rise in malware and attacks that target personal information and mobile banking services. The risks associated with such breaches are increased by the widespread use of smartphones for various purposes, including communication and financial transactions. With predicted trends indicating an increase in malware and viruses specific to smartphones, mobile security has become a key concern for cybersecurity experts.

6. Improving Cloud Security Protocols

As more businesses adopt cloud services, it is crucial to have robust security measures in place for both data operations and storage. Even with strong security measures in place, cloud providers can still face vulnerabilities due to phishing scams, malicious software, and user error.

Organizations must have robust security measures in place to protect against data breaches and cyber threats. Effective cloud security methods must include encryption, authentication, and regular patching to ensure the confidentiality and integrity of cloud-based data and applications.

7. Data Breach: A Recurring Issue

Even minor software errors can lead to significant risks, which is why data breaches remain a major concern for individuals and organizations worldwide. Regulations such as the CCPA and GDPR emphasize the importance of robust security measures in enhancing data protection and privacy rights. To mitigate the risks of data breaches, it is crucial to ensure compliance with regulations and implement proactive security measures.

8. IoT Safety

The spread of 5G networks has ushered in a new era of connectivity, particularly with the Internet of Things (IoT). IoT devices are vulnerable to external threats and software defects, despite providing previously unheard-of communication capabilities. 5G design is still in its infancy, so identifying and addressing any security flaws will require extensive research. The creation of reliable hardware and software solutions must be a top priority for manufacturers to mitigate the risk of network attacks and data breaches.

9. Using Automation to Improve Cybersecurity

To handle the constantly growing amount of data and simplify security procedures, automation is essential. Security professionals can respond quickly and effectively to new threats due to automation, which provides invaluable assistance in the face of high workloads. For large and complex applications, incorporating security measures into agile development methods ensures the production of more secure software solutions.

10. Targeted Attacks Using Ransomware

Industries that rely on specific software systems are at significant risk from targeted ransomware attacks, which can have devastating consequences. Strong cybersecurity measures are crucial, as pointed out by many recent events. To successfully mitigate risks, organizations must be vigilant for ransomware threats and implement preventive measures.

11. Increasing Cyberwarfare

Cyberwarfare and cyberattacks are increasingly targeting sensitive data and vital infrastructure. Elections and other high-profile events are susceptible to cyberattacks. Thus, extra security is required. A spike in data breaches and the exploitation of political and industrial secrets can be anticipated.

12. Mitigating Insider Threats Through Awareness

Mistakes made by individuals continue to play a significant role in data breaches, particularly in cases involving insider threats within organizations. To address this risk, it's vital to enhance awareness and provide thorough training programs for employees. By empowering staff to recognize and address potential vulnerabilities, companies can foster a strong culture of cybersecurity awareness. This approach is crucial for safeguarding sensitive data and minimizing the impact of insider threats effectively.

13. Resolving Issues with Cybersecurity in Remote Work Settings

As employees transition to remote work, they navigate less secure network configurations, creating new cybersecurity challenges. Companies must prioritize implementing robust security measures, such as multi-factor authentication and secure virtual private networks, to effectively protect remote workers from online attacks.

14. Countering Attacks via Social Engineering

Social engineering assaults, such as phishing and identity theft, continue to pose a serious threat to enterprises by taking advantage of human weaknesses to get private data illegally. Proactive security measures and personnel training are crucial in mitigating the risks associated with social engineering attacks.

15. Using Multi-Factor Authentication (MFA) to Increase Security

By requiring users to submit multiple authentication forms before gaining access to accounts or systems, MFA adds an extra layer of security. This proactive strategy strengthens the entire cybersecurity posture and reduces the likelihood of unauthorized access. Adopting multi-factor authentication (MFA) must be a top priority for organizations to properly defend against cyber threats.

16. Preventing International Attacks

Because they target sensitive data and vital infrastructure, sophisticated state-sponsored attackers pose a serious threat to enterprises. Multi-factor authentication and real-time monitoring are two proactive security techniques that are crucial for preventing these sophisticated attacks.

17. Improving Management of Identity and Access

Organizations can monitor and regulate access to networks and sensitive data by utilizing identity and access management (IAM) policies. To protect against unauthorized access and data breaches, robust authentication, authorization, and access control procedures must be implemented.

18. Monitoring Data in Real Time to Identify Threats Early

By enabling enterprises to identify and react quickly to suspicious activities, real-time data monitoring lowers the risk of cyberattacks and data breaches. To detect such risks and mitigate their effects, automated alerts and log monitoring are crucial.

19. Securing Connected Vehicles Against Cyber Threats

The increasing connectivity of vehicles exposes them to cyber threats, necessitating robust security measures to protect against potential attacks. Encryption, authentication, and real-time monitoring are essential to safeguarding connected vehicles against automotive hacking.

20. Ensuring IoT Device Security

Ensuring robust security measures becomes increasingly crucial as the number of Internet of Things (IoT) devices continues to grow. To guard against potential vulnerabilities, organizations must prioritize the security of their IoT devices and implement regular updates and safeguards.

Summing Up

The future of cybersecurity will see advancements in trends focused on enhancing digital fortifications. AI-driven threat detection, zero-trust platforms, and other automated security measures have immense potential in helping mitigate emerging threats.

With remote working environments on the rise, the focus is likely to shift towards securing endpoints while also increasing data encryption measures. Moreover, continuous monitoring, holistic user training, and proactive threat intelligence will become integral in maintaining robust cloud security.

In short, a collective effort involving industry collaboration, the latest solutions, and adaptive strategies will become increasingly important in ensuring a resilient and well-protected cloud security system.