The virtual warehouse in snowflake goes through various states. The states are cold warehouse, warm warehouse, and Hot warehouse. In this article, we will learn about what does each state of virtual warehouses in snowflake. We will also see what impact of states of virtual warehouses on query performance.
A) What is a virtual warehouse in snowflake?
The virtual warehouse in snowflake is a cluster of computing resources that are used to perform activities such as DML operations and SQL execution. The compute resources consist of temporary storage, memory, and CPU. The majority of expenses occur due to the use of a virtual warehouse or compute layer.
B) What are the states of the virtual warehouse in snowflake?
The virtual warehouses go through various states and these are -
a) Cold virtual warehouse
b) Warm virtual warehouse
c) Hot virtual warehouse
In the next section, we will more about all these states of the virtual warehouse.
C) Cold virtual warehouse :
Starting a new virtual warehouse without using local disk cashing and executing the query is called a cold virtual warehouse. When our virtual warehouse is suspended or not active and if we run the query then it will start a new instance of the virtual warehouse.
D) Warm virtual warehouse :
It is the state of the virtual warehouse during which the virtual warehouse is active and running for a while and processed queries. Assume that, the virtual warehouse is in a warm state and we disable the result cache and execute the query then it will use local disk caching. This caching is called warm caching.
E) Hot virtual warehouse :
The hot virtual warehouse is a state of virtual warehouse during which the virtual warehouse is active and running for a while and processed queries. However, in this case, the result cache is on. If we execute the query in this state, the query result is returned from the result cache. This is the most efficient operation.
F) Impact of virtual warehouse state on query performance
1) Cold warehouse
The query takes longer than a warm and hot virtual warehouse. It uses a remote disk. Local disk cache is not used. Result cache is not used.
2) Warm warehouse
The query takes lesser than a cold warehouse but more than a hot virtual warehouse. It does not use a remote disk, however, it uses a local disk. It does not use result cache.
3) Hot virtual warehouse
The query takes lesser time for execution than a cold virtual warehouse and a warm virtual warehouse. It does not use both remote disk and local disk cache. The result is returned using the result cache. from the cloud services layer. It is the most efficient way of getting the result of the query.
Q1. What are the types of data warehouses? What type of snowflake data warehouse is it?
Before understanding the types of data warehouses, we need to know what is a data warehouse? A data warehouse is a central data repository used for data analyses and reporting .
Following are the types of data warehouses
a) Enterprise Data Warehouse (EDW) which is a centralized warehouse used for decision making across enterprises. EDW is used for tactical and strategic decision purposes.
b) Operational Data Store (ODS) which is a centralized database that is the complementary element to EDW and often acts as a source to EDW, ODS gets refreshed in real-time and used for operational reporting and decision making.
c) Data Mart is a subset of a data warehouse and is normally used by a specific team or business line.
Snowflake is an analytic data warehouse i.e can be used as Enterprise Data Warehouse and it is implemented as a software As A service i.e SaaS service.
Q2. Is it possible to use data from the local system to load in Snowflake?
No, we can not load from the local system we need to use Amazon S3 bucket or Microsoft Azure BLOB, or Google cloud storage.
Q3. What are the important features of Snowflake?
The listed below are the features of the snowflake -
1. Database and object closing
2. External Table
3. Geospatial data support
4. XML support
5. Cashing
6. Search optimization services
7. Integration with Hive meta store
8. Data protection and security
9. Time Travel
10. Data sharing
Q4. Can we use an external database such as Oracle or DB2 for Snowflake storage layer ?
No, we can not use an external database for snowflake storage layer snowflake comes with an inbuilt database which is built on SQL database. It is a columnar stored relational database. The snowflake database works well with Tableau, Extel, and many other tools. Snowflake database provides all the services which come with SQL database such as role-based security, query tool, multi-statement transactions, etc.
Q5. What are the cashing areas in snowflake architecture?
The data fetched from the storage layer is cashed at two locations 1) Compute layer 2) Cloud services layer
If cloud services layer cashing is disabled then compute layer cashing is used
This is the first article on a series of Interview Questions and their answers on Snowflake. Through these questions and answers we will learn more about Snowflake so let's start.
Q 1 . What is the architecture of Snowflake?
Snowflake architecture is a hybrid architecture of shard-disk and shared-nothing database architectures. As like shared-nothing architecture, snowflake processes queries using massively processing compute clusters where each node store some portion of data locally. on other hand, as like shard-disk architecture, snowflake uses a central repository for data & it is accessible from compute nodes.
Snowflake architecture has three layers i.e database storage to store data, Query processing or compute layer for processing queries and the third layer is cloud services which provide services such as security, metadata, and optimizer.
Q2. What are cloud platforms are supported by Snowflake?
Snowflake supports the following cloud platforms -
1. Amazon Web Services (AWS)
2. Google cloud platform (GCP)
3. Microsoft Azure ( Azure)
Q3. Do you consider snowflake as an ETL tool?
Yes, snowflake can be considered an ETL tool because it performs extract, transform, and loads operations like other ETL tools.
a) Extract Process: With help of this process, snowflake extracts data from the source and creates data files. these data files support various data formats like CSV, XML, JSON, etc.
b) Transform Process: We can write a custom transform process in snowflake to pull data from the source and apply some transformations for cleansing and standardization and then load the data.
c) Load Process: With the load process, we can load data to the internal or external stage. we can use Microsoft Azure Blob, Amazon S3 bucket or snowflake managed location for staging data. The data is copied to snowflake storage using the COPY INTO command.
Q4. What are the different Editions of Snowflake?
Here is list of snowflake Editions
a) Standard Edition
b) Enterprise Edition
c) Business -critical Edition
d) Virtual Private snowflake
Q5. What kind of SQL does snowflake use?
Snowflake uses ANSI SQL which is a common standard version of SQL.
Are you looking for the details about various phases in BES External calls? Are you also interested in knowing what are categories under which these phases come? If so, then you reached the right place. In this article, we will learn about BES External calls and their phases.
A) What BES External calls?
BES External calls are the web service configured in the provisioning tool in order to achieve customization in Entity 360 or Customer 360 application. we need to develop a custom web service and deploy it on the application server. Once it is deployed, we can register the endpoint URL in the provisioning tool in Informatica MDM.
B) Categories of phases of BES External calls?
The phases of BES External calls are categorized as -
a) Read calls
b) Write calls
c) Merge calls
a) Read calls: This category is used for read and search operations.
b) Write calls: This category uses write co for insert, update and delete operations.
c) Merge calls: This category uses preview merge co which is designed for merge-operations.
C) Phases of External calls
Let's understand detailed phases for the above categories
Are you looking for details about Dynamic Data Masking? Are you also interested in knowing what are the things we need to consider for implementing Dynamic Data Masking also known as DDM? If so, then you reached the right place. In this article, we will explore various aspects of Dynamic Data Masking.
A) What is Dynamic Data Masking (DDM)?
Dynamic Data Masking is a technology using which we can mask production data in real-time. Dynamic Data Masking also called DDM does not change data physically. DDM just changes the data stream in order to mask the sensitive data when the requestor request such information.
B) What are the Dynamic Data Masking tools?
Various vendors provide Dynamic Data Masking functionalities and these are
1. Microsoft Azure SQL Database
2. Oracle Enterprise Manager
3. iMask
4. Informatica Dynamic Data Masking
5. Imperva Data Masking
6. Infosphere Optim Data Privacy by IBM
7. K2 view Data Masking
8. Mentis
C) What are data masking Rules?
The rules contain various conditions and actions that rule engines use in order to process the request.
e.g.
1) Connection rules : It process application connection requests.
2) Security rules : It process SQL statements.
Here are important points about rules
a) We can define and create rules in order to process SQL requests that are executed by an application against the database.
b) DDM rule uses two techniques i.e connection criteria and masking techniques.
c) In order to forward the requests the rule Engine uses connection criteria.
d) In order to mask the data the masking technique is used.
D) What are DDM rule components?
The DDM rule components are as below
a) Matcher: It defines the criteria for the rule engine to identify the match.
b) Action: It defines action which will be applied by the rule engine to request.
c) Processing Action: The rule engine applies specific action to the request after applying the rule.
Do you know why snowflake is one of the leading cloud data warehouse platforms available in the current market? Are you also aware that how Snowflake has evolved over the period of time and still evolving and replacing traditional data warehouses? We are going to understand all these things in this article.
Introduction:
In this article, we are going to see the golden age in which we are living. we will also see the design for the traditional data warehouse. Then we will see how snowflake evolved over the number of years and then finally we will see what's making snowflake a leader in the cloud data warehouse market.
Golden Age:
Let's start with the Golden age. Currently, we live in the golden age of distributed computing. The public cloud platforms such as Amazon web services, Google cloud platform, or Microsoft Azure provides unlimited storage and compute resources and these resources are available on demand. Because of that only end-user can Enterprise-class experience for systems or applications with help of software as a service or SAAS model. For this experience, we do not have to spend a lot of money. These services are cost-efficient and perform well. Cloud Dataware leverages these features but not the traditional data warehouse.
Before going to see what are the drawbacks of the traditional data warehouse implementation. Let's have a look into the design of the traditional data warehouse. As we can see in this screen, the traditional data warehouse has multiple layers, Those are the data source layer, staging layer, warehouse layer, data mart layer, and client layer.
The data source layer brings the data from various sources such as Salesforce, CRM, Human Resources, etc. such data is stored in a traditional database or flat-file format, The ETL i.e. Extract transform, the load is implemented to pull data from source systems and push to staging layer. After performing cleansing standardization data is then loaded from the staging layer to the data warehouse. Along with Raw data, we also store metadata and a summary of data in the data warehouse. Finally, this data is published to the data mart. The data mart night be for sales, inventory, or for purchasing. On top of this layer data mart, the client layer will be present. The business users or business analysts will perform various operations in order to carry out in-depth data analysts, prepare the reports and perform data mining. All these users will connect to multiple data marts for their needs.
As we can see this traditional data warehouse model is complex and resource extensive traditional data ware model is designed considering the fact that it will deal with fixed resources. That was true earlier but with evolution technologies, social media, and advancement in the sector fixed resources design is no more relevant. We deal with a variety of data coming with different speeds and formats. Traditional data warehouses face challenges in managing all these aspects of data.
Another aspect of a traditional data warehouse is investment cost. we need to invest a big chunk of money in the early stages of data warehouse implement which is not the case with the cloud data warehouse.
Complex ETL pipelines are another drawback of the traditional data warehouse. As we can see we need to build multiple pipelines to push data from data source to staging layer, staging layer to the data warehouse, and data warehouse to data mart. Adding flexibility to this flow is a very challenging thing for the traditional data warehouse. Hence snowflake comes into the picture.
Snowflake Evolution:
We are going to see various things about snowflakes but before going to see features and the advancement in the snowflake, let's understand a few things related to snowflakes. currently, snowflake supports three cloud platforms. And those are amazon web services, google cloud platform and Microsoft azure snowflake supports various regions across the world and those are north America, Europe, Asia specific. Each of these regions is supported by respect cloud platforms i.e. Amazon web services, google cloud platform and Microsoft azure.
Let's have look at how snowflake evolved over the period of time snowflake was founded in the year 2012 and was published in Oct 2014. In the same year, it come with the Amazon S3 platform once it become more stable, it introduced Microsoft azure cloud in 2018, and in this year 2019 Google cloud platform was introduced. As we can see, within a short period of time this product has evolved a lot. It will support three major cloud service providers, i.e. Amazon, Google, and Microsoft. As it progresses, it will support many more cloud providers in the future. Snowflake was number one rank in cloud 100 in this year 2019. Snowflake is one of the leading tools in the cloud data warehouse market.
So, what makes snowflake a leading platform? The critical aspect about snowflake is it segregated storage and compute layer traditional data warehouse either support shared-nothing architecture or shared disk architecture. On other hard snowflake brought a hybrid approach on the table with benefits from both shared-nothing & shared disk approach. Apart from it, the snowflake is a pure software as a service, product .i.e users don't have to worry about software installation, administration, product upgrades, etc. It also supports ASSI SQL and ACID transactions. semi-structured data which is difficult to manage with a traditional data warehouse is easily managed & maintained using a snowflake cloud data warehouse. The Elastic storage of computing resources can be scaled independently and seamlessly. That's a very critical aspect brought by a snowflake. It is highly available and it is durable. And of course, it is cost-efficient snowflake is also working on improving cost efficiency furthermore. Last but not least snowflake is secure and comes with end-to-end encryption.
Because of all those features snowflake is the leader in the cloud data warehouse.
Are you looking for details virtual warehouse? Are you also interested in knowing what role virtual warehouse plays in snowflakes? If so, then you reached write place, in this article we will learn about virtual warehouses in snowflake in detail.
A) What is Virtual Warehouse?
Before knowing about virtual warehouse we need to know what is EC2? EC2 is also known as Elastic compute cloud which is a web service that provides secure, resizable compute capacity in the cloud. Now, we know what is EC2, Let's understand what is the virtual warehouse.
The virtual warehouse is an important layer in snowflake architecture and it consists of clusters of EC2 instances. The virtual warehouse is an abstraction by which each cluster is presented to a single user.
B) What is a worker node in the virtual warehouse?
As we know virtual warehouse consists of a cluster of EC2 instances. The individual EC2 instance is called a worker node that performs given tasks. End-user never interacts with worker node. When users perform any action which involves virtual warehouse processing during such time users do not know how many worker nodes are working on and how they are performing tasks in the warehouse.
C) What are the virtual warehouse sizes?
Virtual warehouses come with T-shirt sizes .currently available VW size are
i) x- Small vi ) 2x - Large
ii) Small vii) 3x - Large
iii) Medium viii) 4x - Large
iv) Large ix) 5x - Large
v) x - Large x) 6x - Large
D) Elasticity and Execution Engine
Virtual warehouse comes with two important concepts Elasticity an Execution Engine . The VWs are compute resources and these can be created , resized and destroyed at any point of time . This feature is Elasticity and has no effect on the state of persistant store or database .
The execution engine is implemented by a snowflake and it is a SQL execution engine. This engine is built based on the below features
Are you looking for an article that explains various processes in Informatica Master Data Management (MDM)? If so, then you reached the right place. In this article, we will learn about various processes through which records are loaded to the MDM system. Let's start.
Informatica MDM contains the various processes and those are -
Step 1: The land process transfers data from a source system via ETL jobs to
landing tables in the MDM ORS (Operational Reference Store).
Step 2: The stage process (Stage Job) reads the data from the landing table,
cleanses the data if applicable, and moves the cleansed data into a staging
table via mapping in HUB Console.
Step 3: The load process (Load Job) loads data from the staging table into the
corresponding base object in MDM ORS.
Step 4: The tokenize process (Tokenization Job) generates match tokens based on
match columns that are used subsequently by the match process to identify
candidate base object records for matching.
Step 5: The match process (Match Job) compares two records for points of
similarity. If sufficient points of similarity are found to indicate that the two
records are probably duplicates of each other, then Informatica MDM Hub flags
those records for consolidation.
Step 6: The consolidate process (Merge Job) merges duplicate records into a
single record after duplicate records have been identified in the match
process.
Step 7: Publish
or distribution process is the main outbound flow for Informatica MDM Hub.Hub
integrates with external systems or DB Schemas to share the consolidated
(Golden) Records.
Are you looking for details about HTTPS protocol? Are you also interested in knowing the differences between HTTPS, SSL, and TLS? If so, then you reached the right place. In this article, we will learn more about HTTPS, SSL, and TLS. Let's start.
A) Understand the speed of the data
The data sent over the internet is very fast. It is faster than traditional channels such as wires, optic fiber, air. It will not be an exaggeration if we say data send over the internet with speed of light. Even speed of data sent over the internet is fast, but it still has to go through multiple devices during its journey over the network and that is where criminals target data.
B) What is HTTPS?
The internet consists of distributed client and server information systems. When we access any application using a computer or mobile or any other type of device, ( these devices act as the client ) we send the request to the server. The server can accept or reject the request. If the request is accepted then a connection is created over a specific protocol. In order to establish a communication set of rules which are implemented with the protocol.
HTTP stands for Hypertext Transfer Protocol used on the worldwide web(www). This commonly used protocol defines
1. How data is formatted
2. What type of data is to be transmitted
3. How the server should respond to the specific command
However, HTTP is not secure as it does not have data encryption and authentication functionalities. In order to achieve security especially transmitting data over the network, Hypertext Transfer protocol secure (HTTPS) protocol can be used.
Though HTTPS is a safer solution for the client and server models, this added security isn't automatic. In order to maintain security standards, we need to purchase SSL/TLS certificates from a trusted certificate authority.
C) What is SSL?
The SSL stands for Secure Socket Layer. The internet connections are maintained safely by SSL encryption and decryption method. These connections can be between client to client or client to server or server to server. As SSL is an older protocol, the updated TLS was released in 1999 and it is being commonly used nowadays.
D) What is TLS?
TLS stands for Transport Layer Security. TLS is a cryptographic protocol used for achieving better privacy, data integrity, and authentication compared to SSL. It supports stronger, secure cipher suites and algorithms.
TLS is more commonly used in computer networks, web browsing, instant messaging, email etc.
Would you like to know how does TLS or SSL decryption work? Would you be also interested in knowing Symmetric and Asymmetric cryptography? If so, then you reached the right place. In this article, we will explore decryption with TLS or SSL. Let's start.
A) What TLS or SSL?
As discussed in what is the difference between HTTPS, SSL, and TLS ? article, TLS or SSL is a cryptographic protocol for achieving privacy, data integrity over the network.
B) How does TLS/SSL decryption work?
The TLS and SSL both use asymmetric cryptography. TLS /SSL provides reliable security with high performance.
a) Symmetric Cryptography :
Symmetric cryptography uses a secret key to encrypt data. The generated secret key is shared with the sender and receiver. The secret key should be 128 bits in length in order to achieve security.
b) Asymmetric Cryptography :
Asymmetric cryptography uses private and public keys. The public and private keys are mathematically designed. It requires higher bandwidth. The key length should be a minimum of 1024 bits.
c) Secure session key :
The secure session key is generated by SSL /TLS by using asymmetric cryptography. The secure session key is used to decrypt and encrypt the data transmitted over the network. secure session the TLS handshake is achieved with the secure session key.
C) What TLS handshake?
The TLS handshake is a process to achieve communication between server and client to achieve the below Functionalities -
Are you interested in knowing what is the use of multimerge and merge APIs? Are you also would like to know the difference between merge and multimerge API? If so, then you reached the right place. In this article, we will learn about these APIs in detail.
A) What is Multimerge API?
The Multimerge API is used to merge the list of records together. Multimerge is the generic form of merge API.
B) What is Merge API?
The merge API is used to merge two base object records that are identified as the same base object record.
C) What are the differences between Multimerge and Merge API?
1) Number of records to merge :
a) Merge API allows only two records to merge
b) Multimerge API allows more than two records to merge.
2) Parameters to request :
a) Merge API accepts sourceRecord key and targetRecord key as parameters in the input
b) Multimerge API accepts multiple record key lists as parameters in the request.
3) Consolidated records :
a) Merge API allows records irrespective of the value of consolidation indicator
b) Multimerge API allows merging of unconsolidated records only i.e. consolidation indicator !=1
4) Final value for consolidation indicator :
a) The final value for consolidation indicator after performing merge API operation is 1 i.e consolidated state
b) Multimerge API does not change consolidation indicator value for surviving records.
5) Surviving Record :
a) The surviving record is specified in merge API with targetRecordkey as the parameter.
b) For Multimerge API, the surviving record will be determined based on survivorship rules of the XREF that are participating in the merge process.
Learn more about Informatica MDM survivorship rules here
Are you looking for details about Time Travel in Snowflake? Are you also interested in knowing what are tasks we can perform using Time travel feature? If so, then you reached the right place. In this article, we will learn one of the powerful features is Snowflake.
A) What is Time Travel in Snowflake?
The feature by which we can access historical data at any point within a specified period is called Time Travel in snowflake we can access data not only changed but deleted as well.
B) What are the tasks that can be performed using Time travel in Snowflake?
The tasks below can be effectively performed by using Time Travel Feature
1. Backing up the data from key points in the past.
2. Duplicating the data from key points in the past.
3. Restoring tables, schemes, and databases if those are accidentally deleted.
C) What is Data Protection Lifecycle?
In snowflake, there are three-phase of the data protection lifecycles.
1. Current Data Storage: on the current data set we can perform standard operations such as DML , DDL etc.
2. Time Travel Retention: The normal retention period is 1 to 90 days. Here is the list of operations allowed with time travel.
a) SELECT .... AT| BEFORE ...
b) CLONE ... AT|BEFORE ...
C) UNDROP...
3. Fail safe: This is the last phase in Data Protection Lifecycle. This can only be performed by snowflake No user operations are allowed.
D) Data Retention Period in snowflake
In snowflake, Data Retention Period is a key component for Time Travel. The Data retention period specifies the period or number of days we can preserve data. Snowflake Preserves the state of data before update /delete/drop.
For Snowflake Standard Edition, the Data Retention period is one day.
For Snowflake Enterprise Edition, Data Retention Period between 0 to 90 days.
Are you interested in knowing what are the new featured introduced in Java 17? Are you also interested in knowing what are the deprecated features in Java 17? If so, then reached the right place. This is the second part of the feature in java 17. You can access the first part of the features of Java 17 here.
A) Introduction
In the previous article, we explored the Java 17 features such as JEP 412: Memory API and Foreign Function, JEP 411: Deprecate the Security Manager, JEP 414: Vector API, JEP 415: Deserialization Filters.
In this article, we will focus on the features below in Java 17
1. JEP 409: Sealed classes
2. JEP 406: Pattern Matching for switch
3. JEP 403: Strongly Encapsulate JDK internals
4. JEP 398: Deprecate Applet API for removal
B ) JEP 409: Sealed classes
A sealed class that restricts other classes may extend it. This also applies to interface as well i.e a sealed class can be an interface that restricts another interface may extend it.
With Java 17, new sealed, non-sealed character sequences are introduced and it allows them as contextual keywords.
C ) JEP 406: Pattern Matching for switch
With this change, all existing expressions and statements compile with identical semantics. It performs then without any modification.
There are two new patterns are introduced
1. Guarded Pattern: It is used to refine the pattern matching logic using a boolean expression
2. Parenthesized Pattern: It is used to get rid of parsing ambiguities
D ) JEP 403: Strongly Encapsulate JDK internals
All the internal elements of JPK are strongly encapsulated. Here only exception is sun.misc.unsafe.
E ) JEP 398: Deprecate Applet API for Removal
As we know Applet APIs were deprecated since Java 9 but these were never removed. With Java 17, these will be removed there not be much impact because these Applet APIs are no longer in use as we use more advanced web technologies for it.
Are you looking for detailed information about all the interesting features introduced in JDK 17? Are you also would like various terminologies such as LTS or JEP? If so, then you reached the right place. In this article, we will explore new features in JDK 17 release.
A) What is LTS in Java?
LTS is an abbreviation for Long-term Support. It is a product life cycle management policy. With this policy, the software edition is supported longer than the software standard edition.
B) What is JEP in Java?
JEP is an abbreviation for JDK Enhancement Proposal. Oracle Corporation has drafted this process to collect proposals for enhancements to the Java Development kit i.e. JDK.
C) What are the new features in Java 17?
Java 17 is one of the major releases and comes with various interesting features. In this article we will explore the features below :
1. JEP 411: Deprecate the security manager
2. JEP 412: Memory API and Foreign Function
3. JEP 414: Vector API
4. JEP 415: Deserialization Filters
1 . JEP 411: Deprecate the security manager
The security manager API which was used to define security policy for an Application is deprecated with JDK 17 release. The security manager is deprecated as this API is not commonly used. one of the basic features of a security manager is a blocking system:: exit. If applications continue to use the security manager then an alert message will be issued.
2. JEP 412: Memory API and Foreign function
With JEP 412 the new API is introduced and these are Foreign Memory Access API and the foreign linker API with these API'S we can invoke code outside of the JVM and also security access foreign memory Here, foreign memory means the memory which is not handled by JVM.
3. JEP 414: Vector API
These Vector APIs are part of JDK 16 are also enhanced in JDK 17 to express vector computations on supported CPU architecture at runtime. These are reliable for compilation and performance on AArch 64 and x64 architectures.
4. JEP 415: Deserialization Filters
With a JVM-wide filter factory, we can allow applications to configure context-specific and dynamically selected deserialization filters. This will be helpful to prevent serialization attacks.
Are you looking for details about monitoring Errors in the Alert log? Would you also like to know about ORA errors such as ORA-7445 , ORA-1653 ,ORA-1650 etc? If so, then you reached the right place. In this article, we will understand monitoring Errors in the Alert logs.
A) What are Alert logs?
The important information about error messages and exceptions which occurs during various operation database is captured in the log file called Alert logs.
Each Oracle database for windows instance has one alert log.
B) What is the location of Alert logs?
We set the path for DIAGNOSTIC_DEST initialization parameter. At this path location, the alert log file is created. Normally, the alert file name is alert _SID.log
C) Database crash errors
These errors are associated with an error that can be severe enough to crash an oracle instance. To analyze the oracle instance crash issue we need to capture a trace file or a core dump file and sent it to oracle technical support.
D) ORA - 600 Errors
The ORA-600 will not crash the oracle database. However, it may produce a core dump or trace file -
Example of trace file -
Errors in file /ora/home/dba/oracle/product/rdbms/log ora_ 123.trc
If a data block is read that appears corrupt in such case ORA-1578 is returned. This error message provides details of the file and block number.
e.g.
ORA-D1578 ORACLE data block corrupted (file#xyz, block#01)
F) ORA-1650
It is an error message related to the rollback segment. The error message 'ORA-1650 cannot extend rollback segment ' is produced when the rollback segment has become full. The oracle instance will not crash but the task will be terminated.
e.g. ORA-1650 is unable to extend rollback segment PQR by 64000 in tablespace ROLLBACK.
Based on the above critical error messages we can build the monitoring system.
Are you facing challenges while handling NULL values in snowflake? Are you also interested in knowing what are the things we need to consider while columns in the query contain NULL values? In this article, we will learn details about handling NULL values.
A) What are NULL values?
Many times NULL value is referred to as no value. some systems do not have NULL concepts. In technical terms, the NULL value is a reference to an empty area of the memory. some systems handle differently than others. Snowflake has its own way of handling NULL values.
B) What are the rules for handling NULL values in Snowflake?
Here is the list of rules -
1. Not null to null value comparison :
we compare not null value with the null value it returns NULL
e.g 'ABC ' =NULL returns NULL
2. Equality or inequality comparison :
If we use inequality comparisons such as less than or greater than it results in a NULL value.
e.g 'ABC' > NULL returns NULL
3. Comparision of NULL values :
If we compare one NULL value with another NULL value then it results in a NULL value.
e,g . NULL= NULL results NULL
4. Best way of comparing NULL values is
IS NULL or IS NOT NULL
5. Function to null safe equality
We can use the EQUAL_NULL function to check NULL safe equality.
e.g. equal_null ('ABC' , null) will return false.
6. Aggregate function dismisses NULL values.
Assume that one of the fields in the snowflake contains the below values -
1,2, NULL , 3, NULL 6, 1
If we use aggregate function 4 AVG on this field then it will return the result as 3 i.e(1+2+3+6)/4
i.e. It ignores NULL values.
7. Real average value
In case we need a real average in the above example we need to use the SUM function, where the NULL value is 0
i.e. SUM(<value>)/count (*)
e.g. sum(1+2+0+3+0+6)/6
i.e. 12/6 = 2
8. The empty string in snowflake is NOT NULL
e.g. "IS NULL will returns false.
9. Count (*) returns the total count of rows in the table
10. Count (<column_name>) returns the count of rows with not null values only
Are you looking for an article on snowflake architecture? Are you also looking for the components of snowflake architecture? If so, then you reached the right place. In this article, we will explore database storage, Query processing, and cloud services in detail.
A) What is the Architecture of Snowflake?
Snowflake architecture is a hybrid of a shared-nothing database and shared disk.
1. Snowflake uses a central data repository which is similar to shared-disk architecture.
2. Snowflake processes queries using massively parallel processing compute clusters. In this kind of architecture each node in the cluster, stores a portion of the entire data set. This is similar to shared-nothing architecture.
B) What are the components of Snowflake Architecture?
The components of snowflake architecture are as below
1. Database Storage
2. Query Processing
3. Cloud Services
Let's understand each of these components one by one
1. Cloud Services
It is the topmost layer in snowflake architecture. It consists of a collection of services that coordinates various activities across the Snowflake platform. The cloud services join various components of a snowflake in order to fulfill requests such as login or giving a response back to the user.
Here is the list of services that are handled in this layer.
1. Authentication
2. Infrastructure Management
3. Metadata Management
4. Query Parsing
5. Query optimization
6. Access Control
2. Query Processing
In this layer, query execution is handled. It is the most common and widely used component of the snowflake. The queries are processed using a virtual warehouse. Each virtual warehouse is massively parallel processing compute cluster. It consists of multiple compute nodes provided by snowflake from the cloud provider.
3. Database Storage
It is cloud storage where optimized data is stored. What is optimized data? The optimized data is nothing but the data which is reorganized by snowflake into the compressed and columnar format.
What are aspects handled by snowflake related to data? here is a list which is taken care of by snowflake
1. File Size
2. Structure of the data
3. Compression of the data
4. Metadata
5. Statistics of the data
6. Organization of the data
The important thing here is the data stored by snowflake is not visible or accessible directly by customers. It can only be accessed using SQL query operations.
Are you looking for an article that lists currently available applications which use Artificial Intelligence? If so, then you reached the right place. In this article, we will explore the applications which leverage the benefits of Artificial Intelligence.
1) Google
Google has a predictive search engine that predicts the next word when a user types a keyword to search on the Google page. This recommendation suggested by Google search is one of the best examples of Artificial Intelligence aka AI. It uses predictive analysis to achieve it.
2) JP Morgan chase's contact Intelligence platform
Artificial intelligence, machine learning, and image recognition is used to implement JP Morgan chase's contact Intelligence platform to analyze legal documents. This system is very efficient compared to the manual review of each and every legal document.
3) IBM Watson
It is another Implementation of AI. IBM Watson technology is used by Healthcare organizations for medical diagnosis.
4) Google Eye Doctor
The condition called diabetic retinopathy which can cause blindness can be diagnosed by using this AI-based technology named Google Eye Doctor
5) Facebook
It is one of the social media platforms which uses artificial Intelligence for face verification. Internally it uses machine learning and deep learning to detect facial features and tag friends.
6) Twitter
Twitter uses Artificial Intelligence to detect hate speeches and terroristic languages in the twits.
7) Siri or Alexa
these virtual assistance devices use Artificial intelligence for speech recognition.
8) Tesla
Now a day we hear the buzzword autonomous driving or self-driving cars. Tesla is the leader in it Tesla uses computer vision, image recognization, deep learning in order to build smart cars. Which detects obstacles and drive around them without human interaction.
Are you looking for an article on what are types of artificial intelligence? Are you also interested in knowing what are stages of artificial intelligence are? If so then you reached the right place. In this article, we will focus on various types of artificial intelligence.
A) what is Artificial Intelligence or AI?
The System which is capable of performing tasks that require normally human intelligence e.g decision making, object detection, complex problem solving, etc is called an Artificial Intelligence system, and capability with which it performs is called Artificial Intelligence.
B) Stages of Artificial Intelligence
The stages of artificial intelligence are
1. An Artificial Narrow Intelligence (Weak AI)
Artificial narrow Intelligence is also called weak AI. It is a stage of AI that involves machines that can perform specific tasks.
e.g Alexa or Siri in iPhone
2. Artificial General Intelligence
Artificial General Intelligence is also known as stage AI. In this stage, the machine will possess the ability to think and make decisions.
There is no implementation of strong AI yet.
3. Artificial super Intelligence
It is a stage of AI when the capability of computers will surpass human beings.
This is still considered a hypothetical situation.
C) Types of Artificial Intelligence
The types of Artificial Intelligence are as below-
1. Reactive machine AI :
In this, machine operators solely based on the taking current situation and data
e.g IBM chess machine - Deep Blue
2. Limited memory AI
In this, the machine uses post data and its memory to make informed and improved decisions.
e.g self-driving car - Tesla car
3. Theory of mind AI
In this, the human believes and thoughts can be comprehended by considering emotional intelligence in this type of artificial intelligence.
4. Self-aware AI
In this, the machines will have their own consciousness and become self-aware. This type of AI does not exist yet.
Learn more about Artificial Intelligence and data science here
Are you planning to implement Informatica MDM in your project and starting designing a landing table? Are you also interested in knowing the types of landing table designs? If so, then you have reached the right place. In this article, we will explore factors that need to consider while designing a landing table in Informatica MDM.
A) what is the landing table in Informatica MDM?
Are landing tables are the tables where data from the source is loaded in order to process the data and sent through a stage process to cleanse and standardize it. For the stage process, the landing tables act as source and stage tables as targets.
B) Factors to be considered for landing table design
We need to consider the following factors
While designing landing tables in MDM
1. How many source systems are involved
2. What is the volume from each source system
3. Impact of development timelines
4. Maintenance requirements
5. Partition requirements
C) Type of landing table designs
Based on the Information capture in the previous section, we can design landing tables in two ways
1. One landing table for each source
2. One landing table for multiple sources
1. One landing table for each source
If each source is having different types of data ( e.g.one source is customer-centric and another source is Account centric), if the volume in each source is almost equal, or if we have good development and maintenance bandwidth then we can design one landing table for each source.
2. One landing table for multiple sources
If multiple sources are having similar attributes and data types and volume in each source system is low and if we need to expertise the development time then we can design one landing table for multiple source systems.
