Tackling the Bias in Artificial Intelligence: Four Measures to Adopt

Blog

How AI Decision-Making is Improving Enterprise Outcomes

“AI bias” has been a trending hot subject of research and concern for a while. Recently ChatGPT, the latest language learning model released by OpenAI, has become a viral sensation. However, like many AI models before it, bias can be found in its output. AI bias is a phrase used to characterise circumstances in which ML-based data analytics algorithms exhibit bias towards specific categories of individuals. These biases are typically manifestations of pervasive societal biases about race, gender, biological sex, age, and culture.
In AI, there are two types of bias. The first is algorithmic AI bias, often known as "data bias," in which algorithms are trained with biased data. The other type of AI prejudice is societal AI bias. This is where our societal beliefs and conventions cause us to have blind spots or particular expectations in our thinking. Societal bias impacts algorithmic AI bias, but as the latter grows, we see things come full circle.

What can We Do to Address AI Biases?

Here are some of the remedies:

Algorithm Testing in a Real-World Situation

For example, take the case of job seekers. If the data used to train your machine learning system comes from a select group of job searchers, your AI-powered solution may be untrustworthy. While this may not be a problem if you apply AI to similar candidates, it becomes a problem when you apply it to a new group of candidates that aren't represented in your data collection. In this case, you simply ask the algorithm to apply the prejudices it learnt from the first candidates to a group of people where the assumptions may be inaccurate. To avoid this, as well as to discover and resolve these flaws, you should test the algorithm like how you would use it in the real world.

Considering the So- called Counterfactual Fairness

The meaning of "fairness" and how it is calculated are up for debate. It may also change owing to external factors, implying that the AI must account for these changes as well. Researchers have also worked on a variety of approaches to ensure AI systems can meet them, such as pre-processing data, altering the system's choices after the fact, and including fairness definitions in the training process itself. A potential solution is "counterfactual fairness," which ensures that a model's choices are the same in a counterfactual world where sensitive attributes like ethnicity, gender, or sexual orientation have been altered.

Consider Human-in-the-Loop Systems

The purpose of Human-in-the-Loop technology is to accomplish what neither a human nor a machine can do on their own. When a machine confronts a problem, humans must intervene and fix the problem for them. As a result of the continual feedback, the system learns and improves its performance with each consecutive run. Finally, human-in-the-loop leads to more accurate rare datasets of safety and precision.

Change the Way People Learn About Science and Technology

A significant shift is required in the approach to how people are educated about technology and science. It is high time to restructure science and technology education. Science is currently taught objectively, and more transdisciplinary collaboration and educational rethinking are required.
Some concerns should be addressed and resolved on a worldwide scale and other issues should be addressed locally. Every principle and standard, governing body, and people voting on things and algorithms should be verified from time to time. Making a more diverse data collection will not fix the problem. But that is just one factor.

Will Artificial Intelligence Ever be Unbiased?

The answer is both no, and yes. Well, it's feasible, but an impartial AI is only in dreams and probably it will never exist. This is because an impartial human intellect is unlikely to ever exist. An AI system is only as good as the data it gets as input. Assume you can free your training dataset of conscious and unconscious biases regarding race, gender, and other ideological concepts. In such a situation, you'll be able to build an artificial intelligence system that makes objective data-driven decisions.
In short, the fact is that an impartial human mind, as well as an AI system, will never be realised. After all, people are the ones who generate the skewed data, and humans and human-made algorithms are the ones who evaluate the data to find and rectify biases. However, we can overcome AI bias by validating data and algorithms and applying best practices to collect data, use data, and construct AI algorithms.

Why choose Beinex AI & Automation Services

Beinex, in line with industry standards, helps in the adoption and integration of AI & Automation, hands down. Our support program chips in when interventions or inputs are necessary. And we have comprehensive and robust lab-to-industry processes and pipelines that are morally, ethically sound and cutting edge in character.

Experience-induced Agility

Beinex has a talent pool of coveted consultants who are change agents in diverse domains capable of ushering in an organisation-wide transformation in terms of People-Process-Technology-Data. The depth and breadth of their experience adds to agility and brings adaptability to business contexts.

Tool Mastery & Use-case Libraries

The Consultants at Beinex are masters of the tools they operate in. They are well-versed in the range of tools available in the market of which Beinex is a partner to many of them. A robust eco-system of use-case libraries results in minimal turnaround time from a business point of view.

We frequently hear the claim that computers are unbiased. Unfortunately, this is not the case. People are shaped by their upbringing, experiences, and society, and they internalise certain beliefs about the world around them as a result. AI also is not different from it. AI doesn't exist in a void but is made up of algorithms designed and refined by the same individuals - and it tends to "think" the way it's been taught.

Coming Soon: Tableau 2022.1 – What Does It Offer New

List the unused data sources: Data sources imported in a workbook but not used are highlighted by the new feature. The developer can remove these data sources from the data source list to make the workbook faster.
List of unused fields or data columns: Just like the data sources, workbook optimizer also highlights the data columns not used across the workbook. Removing these at the data source level can help improve the overall performance of the workbook.
List of sheets not used in the dashboard: It is a common practise that developers tend to create sheets not used in the final dashboard. This creates unnecessary clutter and makes the dashboard slower. The workbook optimiser feature gives a list of such sheets which the user can delete to optimise the performance of the workbook further.
Highlights lengthy calculations: The feature provides a list of calculations which are too complex and in turn reduce the performance of the dashboard. Simplifying a few of these can improve performance to a great extent.

The user will see a screen like below as part of the workbook optimizer.

Ask Data Phrase Builder: This feature is available on Tableau Server and Tableau Online Add field would look like as shown in the below screenshot:

Customize View Data: This feature is available on Tableau Server, Desktop and Tableau Online This feature enables to reshape the tabular data behind your visualisation in the View data interface. One can create new columns, remove columns from the default view, change the order and sort the data using this feature. This reshaped data can also be exported as csv file to be shared with the team. Change the root table: This feature is available on Tableau Server, Desktop and Tableau Online Managing multiple data tables becomes easier and flexible with this new feature. One can swap any table to be the root table with a single click. This allows one to change the layout of the table quickly, reshape the data with a different root table and delete a specific table without deleting child nodes. For e.g., let us assume a user had to create a data source for an analysis using 3 tables namely ‘customers’, ‘orders’ and ‘returns’. The user creates the data model such that ‘customers’ is the root table followed by ‘orders’ and ‘returns’. But after performing some analysis the user realises that ‘orders’ should be the main root table. In such cases the user would have to re-create the data again from scratch but with the new ‘Swap with root table’ feature user can do the changes with a few clicks. Parameter Enhancements in Tableau Prep In version 2022.1 Tableau Prep is adding even more places where one can use parameters in the flow as well as user enhancements. Now, one can:

Get a list of all the parameters in one place rather than finding them in the flow. One can delete these parameters directly from the parameter window rather than to find it in the flow first.
Include parameter names in exported output files.
Include parameters in SQL scripts that you run before or after writing the flow output to a database. Include parameters in worksheet names when writing the flow output to Microsoft Excel.

New Visualisation Toolbar: Available on Tableau Online and Server The new feature enhances the user dashboard navigation in Tableau Online and Tableau Server. Also, the new toolbar supports embedding for your external applications. However, the available functionality within the toolbar will vary based on the user’s access level. TabPy Model Evaluation Function: This feature is available on Tableau Server, Online and Desktop A new calculation type called MODEL_EXTENSION_X has been added to Tableau specifically to call deployed models from TabPy or other analytics extensions. It gives administrators a new configuration option to control TabPy deployments by disabling the execution of arbitrary Python code. Administrators can now maintain a single source of truth in the Analytics Extension server by leveraging a central version control, while end-users have an improved experience by easily accessing deployed functions on their TabPy server and understanding the model's requirements before adding it to the tool. Apart from the above Tableau 2022.1 has other exciting features the users can look forward to including:

1. Improvements in Esri Data Connector
2. Addition of new Accelerators
3. Added connectors to connect to more data sources

As with the earlier version, Tableau 2022.1 which has addressed a lot of the pain points experienced by its users. Certainly, it has a lot more to offer and enhancements to look forward to. Please download the new version and explore the new features. In case of any challenges while dealing with the new features or in case you need any help related to Tableau development, please reach out to the info@beinex.com. Would be more than happy to help you on the same!

Going Further, Together with AI-driven Sentiment Analysis (Infographics)

Snowflake Time Travel: How to Leverage it to Achieve Optimal Results

Benefits of Snowflake Time Travel

With Snowflake Time Travel, you can access historical data, including data that has been altered or deleted, at any given point. This feature is helpful for various tasks, such as:

• Querying data that has been modified or erased in the past
• Duplicating entire tables, schemas, or databases at or before specific dates
• Restoring deleted tables, schemas, and databases

How to activate Snowflake Time Travel?

Activating Snowflake Time Travel is a simple process that requires no additional effort. It is automatically activated with a retention period of one day. Nonetheless, upgrading to the Snowflake Enterprise Edition is necessary to customise the Data Retention Period and extend it to 90 days for Databases, Schemas, and Tables. It's important to note that increasing the Data Retention Period results in additional storage usage, reflected in your monthly Storage Fees.

Data Retention Period in Snowflake

In Snowflake, Data Retention Period determine how long historical data is retained to support Time Travel functionality. When data in a table is altered, such as through deletions or updates, Snowflake maintains the previous state of the data so that Time Travel operations (like SELECT, CREATE...CLONE, UNDROP) can be performed on it. By default, all Snowflake accounts have a standard retention period of one day (24 hours).

However, the Retention Period can be adjusted at the account and object level in the Snowflake Standard Edition to 0 (or unset to the default of 1 day) for databases, schemas, and tables.

In the Snowflake Enterprise Edition or higher, the Retention Period can be set to 0 for temporary databases, schemas, tables, and temporary tables. For permanent databases, schemas, and tables, the Retention Time can be configured to any duration between 0 and 90 days.

Functions of Snowflake Time Travel SQL Extensions

Snowflake Time Travel SQL Extensions are special SQL commands that allow users to query historical data from a specific point in time using the Time Travel feature. These extensions enable users to perform various Time Travel operations, including:

a. CLONE: This command creates a copy of a table, schema, or database at a specific point in time using Time Travel.
b. UNDROP: This command restores a dropped table, schema, or database to a specific point in time using Time Travel.
c. HISTORY: This command retrieves the history of changes made to a table, schema, or database over time using Time Travel.
d. AS OF: This command retrieves data from a table as it appeared at a specific point in time using Time Travel.

Specifying a Custom Data Retention Period for Snowflake Time Travel

To specify a custom Data Retention Period for Snowflake Time Travel, you can use the DATA_RETENTION_TIME IN_DAYS argument in the command when creating a table, schema, or database. By default, the maximum Retention Time in Standard Edition is set to 1 day (i.e. 24 hours), while in Snowflake Enterprise Edition (and higher), it can be set to any value up to 90 days.

The Data Retention Time can be set in the way it has been placed in the example below.

To create a schema with a custom Data Retention Period of 60 days, you can use the following SQL command:

create table mytable(col1 number, col2 date) data_retention_time_in_days=60;

Modify the Data Retention Period for Snowflake Objects

To modify the Data Retention Period of a Snowflake object, any change made to the Retention Period affects both active data and data in Time Travel. Depending on whether the period is increased or decreased, the following impacts occur:

a. Increasing Retention
b. Decreasing Retention

Let’s dive deep into more details:

a. Increasing Retention
Snowflake Time Travel preserves the data for a more extended period. For instance, if a Table’s Retention Time is increased from 10 to 20 days, the data set to be deleted after ten days will be retained for an additional ten days before being moved to Fail-Safe. However, data over ten days old and already transferred to Fail-Safe mode is unaffected.

b. Decreasing Retention
The duration of data stored in Time Travel is reduced. The shorter Retention Period applies only to active data updated after the Retention Period is shortened. If the data is still within the new Retention Period, it stays in Time Travel; otherwise, it is placed in Fail-Safe Mode. For instance, if a table with a 10-day Retention Period is reduced to 1 day, data from day 2 through day ten will be transferred to Fail-Safe, and only data from day one will be accessible through Time Travel.

Since the background process moves the data from Snowflake Time Travel to Fail-Safe, it may take some time to see the changes. Although Snowflake guarantees that the data will be transferred, it does not specify when the process will be finished. The data remains accessible via Time Travel until the background process is completed. To change an object's Retention Period, use ALTER object command, such as the following command for modifying a table's Retention Period:

alter table mytable set data_retention_time_in_days=30;

Snowflake Time Travel Data Query

To query previous versions of data in Snowflake Time Travel, you can use the AT | BEFORE Clause after making any DML actions on a table. This clause allows you to query data at or before a certain point in the table's history throughout the retention period. The specified threshold can be either time-based (e.g., a timestamp or time offset from the present) or a statement ID (e.g., SELECT or INSERT).

For example, to select historical data from a table as of a specific date and time, you can use a query like:

sql
SELECT * FROM my table AT (TIMESTAMP => 'Fri, 05 May 2023 16:20:00 -

If you want to pull data from a table that was last updated a certain number of minutes ago, you can use a query like:

sql
SELECT * FROM my_table AT(OFFSET => -60*5);

And to collect historical data from a table up to a specified statement's modifications, but not including them, you can use a query like:

Sql
SELECT * FROM my_table BEFORE(STATEMENT => '8e5d0ca9-005e-44e6-b858-a8f5b37c57

How to Restore Deleted Objects by Utilising the UNDROP Command?

To restore a deleted object that hasn't been permanently removed from the system (meaning it can still be seen in the "SHOW object type> HISTORY" output), you can use the UNDROP command in conjunction with Snowflake Time Travel. This command can be applied to various objects, such as tables, schemas, and databases. It effectively reverts the thing to its previous state before it was deleted with the DROP command. For example, the UNDROP command can also restore a dropped database.

Summing Up

Snowflake Time Travel’s features can enhance your decision-making process and overall data experience. If you're looking for a Snowflake service provider, Beinex is an excellent option. Our partnership with Snowflake enables us to offer advanced features like automated tuning, elastic compute, and analytics modernisation services to help your organisation realise exponential Returns on Investment.

SELF SERVICE AUDIT ANALYTICS

TRANSFORM THE WAY YOU AND YOUR ORGANIZATION WORK WITH DATA –SELF SERVICE AUDIT ANALYTICS Increasing complexities of risks and incessant emergence of disruptive technologies are demanding substantial change in internal audit processes. In today’s world of constant disruption, internal audit should evolve into a dynamic and future-oriented function. Businesses need broad-spectrum audit processes that extend beyond reviewing the obvious. Auditors should adopt forward-looking IA approaches, and should be able to provide deeper and valuable insights on strategy, execution, emerging risks, and hidden opportunities. The 2016 Deloitte Global Chief Audit Executive Survey that polled more than 1200 CAEs from 29 countries and a diverse range of industries, reaffirms the growing need to conduct analytics-based auditing. More than a three quarter of the CAEs (79%) recommend the need for digital disruption and innovation to transform internal audit and enhance its value. The survey also cited the increasing relevance of cutting-edge technologies such as artificial intelligence, cognitive computing, and visual analytics. Is skill-gap a concern? More than half of CAEs (57%) who participated in the survey expressed their intense dissatisfaction about inadequate skills and insufficient expertise of audit teams. When left unaddressed, these skill gaps will weaken auditors’ capabilities to deliver on changing stakeholder expectations. Stakeholders expect more forward-looking analysis to uncover risks and hidden opportunities. Or, gone are the days of static audit reports and analysis of sample data. The Deloitte survey also cites risk anticipation (39%) and data analytics (34%) as the two groundbreaking innovations that are most likely to impact internal audit within the next five years. Changing business landscapes, technological advancements, and proliferation of data have brought forth the imminent need to leverage analytics and data visualization to increase the impact, influence, and effectiveness of internal audit Analytics Adoption Challenges Even after 30 years of inception of data analytics, many auditors continue to adopt conventional internal audit methods and lag in technology adoption. Wondering why? Here are some of the reasons:

Skills gap

Insufficient IT support

Difficulty to manage and manipulate data

Increasing requests for ad-hoc analysis and one-off reports

Difficulty in dealing with the basic aspects of data management and governance

Can technology disruption be a savior? Advancements in technology are fundamentally changing the nature of the audit and improving its effectiveness and relevance. Here are a few game-changing technology solutions that auditors can harness effectively to enhance the way they work with data: 1.Self-service analytics – Smart analytics for all Self-service analytics is no more a buzzword. It’s the new norm. Self-service analytics empowers auditors to access data, perform queries, and create interactive reports that add richness and granularity to the insights derived. Anyone and everyone, with or without technical expertise, can harness the power of self-service analytics to work seamlessly with large data sets of any size or type, and discover savvy insights without having to write codes or learn programming languages. Key advantages of using self-service analytics tools: Explore your data and create ad-hoc reports without IT skills

Easy access to any source data

Guided analysis – Faster answers to complex questions

Intuitive drag-and-drop interface to create and share interactive reports.

Natural language processing to respond to complex queries

Interactive visualizations and personalized dashboards to identify patterns and trends

Fast to deploy and easy to manage

All-encompassing data analysis anywhere, at anytime

2.Mobile Analytics – Audit insights on the go Regardless of the size of an organization or availability of data, it takes weeks to prepare and present comprehensive audit reports. The numbers are usually saved offline as large files or copied to multiple slides for boardroom meetings. Mobile data analytics enables a concise and easily accessible digital avatar of audit reports and dynamic dashboards that can be accessed on mobile devices to interact and proactively monitor the business information on the go. 3.Data Visualization When it comes to presenting audit data, reports and findings, one gets a single view of the entire raw data, all at once. This makes it difficult to decipher what is important and what isn’t, the reason why the point of sharing information gets defeated. Data visualization helps get a flexible and reliable way to identify and share pertinent information in a manner that everyone can easily understand.

Process more information than reading numbers

Discover insights using spatial relationships, colors, and textures

Make data accessible to a broader audience and provide users with a rich and engaging experience

There are many reasons why auditing is ripe for self-service analytics and visualization driven transformation. There is more data to examine within limited time span. Most financial and operational transactions are moving online, and the number of variables, outliers, trends, and patterns to identify and analyze continue to increase each day. Visual analytics is the fastest way to analyze and understand structured or unstructured data of any size, without IT assistance. Visual technologies help speed up and improve decision making with heat maps, bubble charts, and interactive dashboards that are easy for C-suite executive, non-technical business users, and stakeholders to understand. Everything gets better when you can do it yourself, right? Self-service audit analytics and visualization too are not different. Benefits of self-service analytics in Internal Audit

Infographics | Cloud Storage and On-Premise Storage: The Tug of War that Never Ends, Ends Now

Amidst cloud storage's ripening, corporate IT departments continue to weigh the pros and cons of on-premise storage vs cloud storage. Before making the right choice for your company, it is always better to analyse the differences between on-premises and cloud-based services and infrastructure.