On my main blog, I broke down:

• The exact SQL problems I was asked (recursive queries, windowing, multi-level filtering)

• Advanced PostgreSQL internals, like execution plans, sharding design, and logical replication

• Real discussions around performance bottlenecks, indexing decisions, and CDC strategies

🎯 Whether you’re prepping for a systems-heavy role or improving your Postgres stack, this walkthrough will help.

👉 Read the full post here

In this blog, we'll explore everything about Microsoft SQL Server In-Memory OLTP—what it is, its use cases, benefits, limitations, and practical examples using the AdventureWorks database. This guide will also cover best practices, monitoring tips, and how to optimize hash bucket counts for memory-optimized tables.

What is SQL Server In-Memory OLTP?

SQL Server In-Memory OLTP is a memory-optimized database engine designed to dramatically improve the performance of transaction-heavy workloads. By keeping tables and indexes in memory and using natively compiled stored procedures, it minimizes latency and maximizes throughput.

Key Features

1. Memory-Optimized Tables: Store tables entirely in memory for ultra-fast access.

2. Natively Compiled Stored Procedures: Execute T-SQL procedures as native machine code.

3. Optimistic Concurrency: Reduce lock contention for highly concurrent workloads.

4. Hybrid Durability: Choose between fully durable (persistent to disk) and non-durable (memory-only) options.

5. Optimized Indexing: Supports hash and range indexes tailored for specific query patterns.

When to Use In-Memory OLTP

Not every situation requires In-Memory OLTP. It's most effective for:

• High-Throughput Applications: Payment systems, stock trading platforms.

• Real-Time Analytics: IoT dashboards, e-commerce order analytics.

• Session Management: Web applications handling large user sessions.

• Caching Layers: Storing frequently accessed reference data.

• Gaming Leaderboards: Fast-paced updates and queries.

Limitations of In-Memory OLTP

1. RAM Dependency: Requires sufficient physical memory to store tables and indexes.

2. T-SQL Limitations: Not all T-SQL features (e.g., triggers, foreign keys) are supported.

3. Durability Trade-Offs: Non-durable tables lose data on server restart.

4. Indexing Constraints: Supports only hash and range indexes.

5. Maintenance Overhead: Requires careful schema planning and memory monitoring.

Hands-On: Implementing In-Memory OLTP with AdventureWorks

Let's get practical and implement In-Memory OLTP using the AdventureWorks database. We'll create a memory-optimized table to cache product data and boost query performance.

Step 1: Set Up the AdventureWorks Database

First, download and restore the AdventureWorks database:

RESTORE DATABASE AdventureWorks
FROM DISK = 'C:\AdventureWorks.bak'
WITH MOVE 'AdventureWorks_Data' TO 'C:\AdventureWorks_Data.mdf',
     MOVE 'AdventureWorks_Log' TO 'C:\AdventureWorks_Log.ldf';

Step 2: Add a Memory-Optimized Filegroup

Memory-optimized tables require a special filegroup:

ALTER DATABASE AdventureWorks
ADD FILEGROUP InMemoryFG CONTAINS MEMORY_OPTIMIZED_DATA;

ALTER DATABASE AdventureWorks
ADD FILE (NAME = 'InMemoryFile', FILENAME = 'C:\InMemoryFile') 
TO FILEGROUP InMemoryFG;

Step 3: Create a Memory-Optimized Table

We'll create a memory-optimized table to cache product data:

CREATE TABLE dbo.MemoryOptimizedProductCache (
    ProductID INT NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 10000),
    Name NVARCHAR(50) NOT NULL,
    ListPrice MONEY NOT NULL
) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_ONLY);

Step 4: Populate the Table

Load data from the Production.Product table:

INSERT INTO dbo.MemoryOptimizedProductCache (ProductID, Name, ListPrice)
SELECT ProductID, Name, ListPrice
FROM Production.Product;

Step 5: Query the Memory-Optimized Table

Run a query to retrieve high-value products:

SELECT * 
FROM dbo.MemoryOptimizedProductCache
WHERE ListPrice > 500;

Step 6: Create a Natively Compiled Stored Procedure

Boost performance further with a natively compiled stored procedure:

CREATE PROCEDURE dbo.UpdateProductPrice
WITH NATIVE_COMPILATION, SCHEMABINDING
AS BEGIN ATOMIC WITH (TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = 'English')
    UPDATE dbo.MemoryOptimizedProductCache
    SET ListPrice = ListPrice * 1.10
    WHERE ProductID = 1;
END;

Execute the procedure:

EXEC dbo.UpdateProductPrice;

Verify the changes:

SELECT * 
FROM dbo.MemoryOptimizedProductCache
WHERE ProductID = 1;

Optimizing Hash Bucket Count

The BUCKET_COUNT in a hash index determines the number of "slots" for rows. Setting it correctly reduces hash collisions and enhances query performance.

Guidelines for Setting BUCKET_COUNT

1. Match Row Count: Set BUCKET_COUNT equal to or slightly greater than the estimated row count. Example: For 1 million rows, use:

    BUCKET_COUNT = 1000000;
   

2. Avoid Over-Provisioning: Too high a BUCKET_COUNT wastes memory.

3. Use Powers of 2: SQL Server performs better with bucket counts as powers of 2.

Monitor Collisions

Use DMVs to check hash index statistics:

SELECT object_name(object_id) AS TableName,
       bucket_count,
       row_count,
       hash_collisions
FROM sys.dm_db_xtp_hash_index_stats
WHERE object_id = OBJECT_ID('dbo.MemoryOptimizedProductCache');

Cleaning and Maintaining In-Memory OLTP Tables

1. Cleaning the Table

• TRUNCATE:

    TRUNCATE TABLE dbo.MemoryOptimizedProductCache;
  

• DELETE:

    DELETE FROM dbo.MemoryOptimizedProductCache WHERE ListPrice < 100;
  

2. Shrinking Checkpoint Files

For durable tables (SCHEMA_AND_DATA), reduce checkpoint file size:

CHECKPOINT;

Monitoring In-Memory OLTP

Use the following DMVs to monitor memory-optimized tables:

1. Memory Usage:

    SELECT * FROM sys.dm_db_xtp_memory_consumers;
   

2. Checkpoint Stats:

    SELECT * FROM sys.dm_db_xtp_checkpoint_stats;
   

3. Garbage Collection:

    SELECT * FROM sys.dm_xtp_gc_stats;
   

Points to Note

1. Plan Memory Usage: Allocate sufficient RAM and monitor usage.

2. Index Smartly: Use hash indexes for equality searches and range indexes for range queries.

3. Choose Durability Wisely: Use SCHEMA_ONLY for non-critical data and SCHEMA_AND_DATA for persistence.

4. Monitor Regularly: Use DMVs to identify bottlenecks and optimize performance.

Summary

SQL Server In-Memory OLTP is a powerful tool for high-performance applications. By using memory-optimized tables and natively compiled procedures, you can achieve unprecedented speed and scalability for transaction-heavy workloads. However, success requires careful planning, proper configuration, and regular monitoring. Implement the techniques in this hands-on guide with the AdventureWorks database and see the difference it makes to your application's performance. Have your insights? Share them in the comments below! For more detailed documentation, visit the official doc’s page.

To make REST API calls to external endpoints of dummy JSON, let's create API resources and add endpoints. Then API Management Service acts as a gateway for the external API calls and provides features such as caching and throttling.

Azure API Management Service Configuration:

When you log into your Azure portal, go to API Management Service, and create a new resource, if you don't have one already. When you open the resource, you will see the APIs tab on the left. When you click on APIs within the APIs tab, you will open a screen on the right where you can create and configure APIs as well as add operations to those APIs. To define a new API, click on the Add API tab and select http on the newly generated screen.

A modal will open as shown below. Keep the type Basic and then provide Display Name and Name. Also, input the Web Service URL, we are using dummyjson for demonstration so input the base URL path of the same which is https://dummyjson.com . After that click on create button.

A new screen will be presented, allowing you to add and configure REST endpoint operations, as well as a Frontend Card for OpenAPI specification editor, Inbound and outbound processing to modify requests or responses, and finally, a Backend Card providing a form-based editor as well as a code-based editor.

By clicking on Add Operation, the Frontend form-based editor will open. Enter the desired details like Display Name and Name. In the URL input field, choose the HTTP method as GET to fetch all product data. Then, enter the endpoint path in the accompanying field, in our case, we added "/products". You may also provide descriptions and relevant tags. Below Description, there are several tabs such as Template, Query, Headers, Request, and Responses that allow you to specify necessary URL template parameters, additional query parameters, important request headers, request body information and define response status codes, content-types and schemas respectively. Finally, save your changes by clicking on the save button.

Then, select the operation we just created and click on the Test tab above to get the HTTP Request URL and Ocp-Apim Subscription Key. You can find them by scrolling down in the HTTP Request Section, inside the code snippet, and clicking on the eye icon.

Click the Send button below to test out the API call and see the response.

We have to connect Azure SQL Database, to call the API from T-SQL code. Below is the code for both the GET method and the POST method. You can add the POST endpoint operation to API Management Service with all the required fields and test it out.

Making External API endpoint call with Azure SQL Database

declare @ret as int, @response as nvarchar(max), @headers nvarchar(4000);
set @headers = N'{"Ocp-Apim-Subscription-Key": "<YourOcpApimSubscriptionKey>"}';

exec @ret = sp_invoke_external_rest_endpoint 
    @method = 'GET',
    @url = 'https://api-management-service-kiran04.azure-api.net/products',
    @headers = @headers,
    @response = @response output,
    @timeout=30;
select @ret as ReturnCode, @response as Response;

```
Below is the example for POST method for reference. You can try adding the POST endpoint operation in API Management Service with all the needed fields and test it out. After that you can try out the POST operation with below example code.
```
declare @ret int, @response nvarchar(max), @payload nvarchar(max), @headers nvarchar(4000)
set @payload = N'{
  "title": "Large Paint Brush",
  "description": "An Large Paint Brush for painting on walls or large surface.",
  "price": 900,
  "discountPercentage": 1.96,
  "rating": 4.69,
  "stock": 94,
  "brand": "ABCXYZ",
  "category": "home appliances"
}'
set @headers = N'{"Ocp-Apim-Subscription-Key": "<YourOcpApimSubscriptionKey>"}';

exec @ret = sp_invoke_external_rest_endpoint
    @method = 'POST',
    @url = 'https://api-management-service-kiran04.azure-api.net/products/add',
    @headers = @headers,
    @payload = @payload,
    @timeout = 30,
    @response = @response output;

select @ret as ReturnCode, @response as Response;

API Management Service is required since we were using external API endpoints for demonstration. As mentioned earlier, this is not required for using other Azure services.

As we normally know Tables provide a static representation of data, displaying only the latest updates. The historical changes or deleted records are lost, posing significant challenges for auditing, analysis, and regulatory compliance. Developers needed to resort to manual techniques, such as triggers and auxiliary tables, to maintain a record of the data's evolution. These methods are cumbersome, error-prone, and lack efficiency.

Hence, Here SQL Server's System Versioned Temporal Table is an apt solution to the limitations mentioned before. As it provides built-in support to automatically track the complete history of data changes without any external script. Each row in the table is associated with two timestamp columns of the datetime2 data type. This timestamp indicates the period during which the particular version of the record was valid. And these timestamps are automatically updated with every modification. The main table that stores current data is referred to as the current table, or simply as the temporal table.

During temporal table creation, users can specify an existing history table (which must be schema compliant) or let the system create a default history table. Some of the use cases are,

• With built-in history tracking of every data modification, auditing and compliance reporting task becomes a breeze increasing transparency and accountability.

• This built-in feature helps to meet compliance standards and ensures adherence to regulatory guidelines.

• It also provides a mechanism to Audit the data changes as the complete history is maintained Recovering from accidental data changes. For instance, if someone has wrongly deleted a record, because of the availability of the history data we can easily recover these deleted records.

Creating Temporal Table

The syntax for an example temporal Table creation is as below,

CREATE TABLE dbo.department
(
  dept_id INT NOT NULL PRIMARY KEY CLUSTERED,
  dept_name VARCHAR(50) NOT NULL,
  manager_id INT NULL,
  parent_dept_id INT NULL,
  valid_from DATETIME2 GENERATED ALWAYS AS ROW START HIDDEN NOT NULL,
  valid_to DATETIME2 GENERATED ALWAYS AS ROW END HIDDEN NOT NULL,
  PERIOD FOR SYSTEM_TIME (valid_from, valid_to)
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE=dbo.department_history));

Few points to remember while creating a System Versioned table.

1. The table must have a Primary Key, otherwise, it would raise the error.

2. The table must also have two DATETIME2 columns which store the start and end timestamps of the row's validity period. The "GENERATED ALWAYS AS ROW START" clause indicates the start time for the row, similarly "GENERATED ALWAYS AS ROW END" clause indicates the end time for the row.

3. The two columns that store the start time and end time for the rows shouldn't be null, otherwise, it will raise an error. Hence, the NOT NULL constraint is used.

4. The use of the HIDDEN flag for the period columns in System-Versioned Temporal Tables makes them effectively hidden from query results, ensuring a clean and streamlined view of the data. And when performing an INSERT INTO statement without specifying the column list, there is no need to provide values for these hidden columns in the values list. It's optional.

5. The PERIOD FOR SYSTEM_TIME(valid_from, valid_to) clause takes the names of the columns which was made for start and end timestamps. SQL Server Engine will use the columns to record the period for which a record is valid.

6. HISTORY_TABLE: This input parameter is used to specify the name of the History Table. And it's optional.

7. SYSTEM_VERSIONING: This argument is used to enable/disable system versioning on the table.

After executing the above statement, if we look into the SSMS window we can see that a dbo.department table was created and it is marked as a System-Versioned table. Inside this table we can see a nested table dbo.department_history table created and marked as History Table as shown in the below image:

DML operations on Temporal Table

Insert Operation:

Let us perform the insert operation to add a new record in the dbo.department table, with the statement below,

INSERT INTO dbo.department (dept_id, dept_name, manager_id, parent_dept_id) values (1, 'HR Department', NULL, NULL);
INSERT INTO dbo.department (dept_id, dept_name, manager_id, parent_dept_id) values (2, 'Sales Department', NULL, NULL);

Since the columns valid_from and valid_to are marked as HIDDEN, we can skip to specify it in the insert statement. If these columns were not marked as HIDDEN, then we would have to pass DEFAULT as values for the columns. Let's check the records in the dbo.department table and dbo.department_history table.

select * from department;
select * from department_history;
select dept_id, dept_name, manager_id, parent_dept_id, valid_from, valid_to from department;

We can see that the insert operation adds new records in the department (Temporal) table and not in the department_history history table. If you want to see the period's columns in the select statement, you need to mention it explicitly in the select statement. Since while creating the Temporal table, we marked these columns as HIDDEN it won't show unless mentioned explicitly like in the third select statement above.

In the image, which is showing the outputs of the select statements of the Temporal & History table also notice the periods columns valid_from and valid_to. The columns hold date-time value, valid_from represents the time of the execution of the insert statement and valid_to represents the maximum value of the DATETIME2 data type. This also indicates that the record is active.

Update Operation:

Now, let's try to update the department table to assign the manager_id value to HR Department and see how it reflects in the main (Temporal) table and History table.

update department set manager_id = 1 where dept_id = 1;

Again perform select queries on both department table and department_history table to check the output.

select dept_id, dept_name, manager_id, parent_dept_id, valid_from, valid_to from department;
select * from department_history;

In the above image showing the outputs, you can see that the update of a record in the temporal table has resulted in inserting the record in its history table with the state of the record before the update and the end time i.e. valid_to column value will be the time at which the update statement is executed. And if you noticed, the temporal table which only holds the current state of the record is updated and the manager_id value is assigned. But the important part is that it has also updated the valid_from column value to the time at which the update statement is executed. The record is still active and has the maximum value of the DATETIME2 data type for the valid_to column.

Important point to note that, you cannot update the PERIOD column values (valid_from, valid_to) when the SYSTEM_VERSIONING is ON. If you need to update the PERIOD columns then first we need to disable the system versioning and then drop the PERIOD definition from the department table, which will make the tables become like any regular table and can perform any operations.

Delete Operation:

Now we will perform a DELETE operation by deleting the record of the Sales department, which has dept_id of value 2.

DELETE FROM department WHERE dept_id = 2;

After executing the above statement, we will perform the select queries like earlier and see how the delete operation has caused any changes in the Temporal table and History table.

select dept_id, dept_name, manager_id, parent_dept_id, valid_from, valid_to from department;
select * from department_history;

As per the results shown in the image just above, we can see that the record for department id 2 which was the Sales department, was removed from the Temporal table, and the state of the record before the delete operation is stored in the History table with valid_to value showing when the delete operation has taken place.

Furthermore, it is important to note that, the TRUNCATE operation on the Temporal Table is not supported like regular tables. And as long as System Versioning is enabled, DML operation on the History table is not supported.

Drop Temporal Table:

To clean up and Drop Temporal and History Tables, we need to first disable the system versioning. And also need to drop the period definition. Below is the script based on our example.

-- disable system versioning
ALTER TABLE dbo.department SET (SYSTEM_VERSIONING = OFF)
GO

-- drop period definition
ALTER TABLE dbo.department DROP PERIOD FOR SYSTEM_TIME
GO

-- drop department and department_history tables
DROP TABLE dbo.department
DROP TABLE dbo.department_history

With this, we wind up the post about the intro for Temporal Table or SYSTEM-VERSIONED Table. To summarize, a system-versioned table, also called a temporal table, is a type of user-defined table to automatically keep track of the full history of data changes or mutations in table rows. Since it tracks the full history of data changes in the separate history table, with the database engine managing validity for each row, helping us to know the state of data at any given point in time in history. In the next post, we will look into Querying the System-Versioned Temporal Table and How to enable System Versioning for the existing regular table and more related topics.

An exception is an error or problem that occurs during the execution of a Transact-SQL statement or batch. When an exception occurs, SQL Server raises an error message that includes information about the error such as the error number, severity, state, and message text.

Some common examples of exceptions in SQL Server include:

1. Constraint violation errors (e.g. foreign key violation, unique constraint violation)

2. Data type conversion errors

3. Divide-by-zero errors

4. Permission errors

5. Object not found errors

6. Deadlock errors

To handle exceptions in SQL Server, we have to use the TRY-CATCH construct. The TRY block contains the code that might raise an exception, and the CATCH block contains the code that handles the exception. By using TRY-CATCH, you can gracefully handle errors and prevent them from terminating your application. In addition to TRY-CATCH, SQL Server also provides other error-handling mechanisms such as RAISERROR and THROW. RAISERROR is used to generate a custom error message and THROW is used to re-throw an exception that has been caught in a TRY block. Handling exceptions is an important part of developing stable and reliable SQL Server applications. Note that the try-catch mechanism won't work in UDF(User Defined Functions).

Understanding the Try Catch block:

The exception handling of the SQL Try-Catch block is the same as the try-catch block in other programming languages. When the SQL statements are executed in the try block, if any exception arises, then the control is immediately transferred to the catch block which handles the flow of the program in case of exceptions. In catch, we write statements or call procedures to properly handle the exceptions that occurred in the try block as per our requirement. We can also take advantage of system functions, to get detailed information about the error.

ERROR_MESSAGE() - you can take advantage of this function to get the complete error message.

ERROR_LINE() - this function can be used to get the line number on which the error occurred.

ERROR_NUMBER() - this function can be used to get the error number of the error. ERROR_SEVERITY() - this function can be used to get the severity level of the error. ERROR_STATE() - this function can be used to get the state number of the error. ERROR_PROCEDURE() - this function can be used to know the name of the stored procedure or trigger that has caused the error.

Syntax:

Below is the syntax for the TRY CATCH.

BEGIN TRY
    -- SQL statements executing queries and batches
END TRY
BEGIN CATCH
    -- SQL statements to handle exceptions
END CATCH

Below is the syntax for the Nested TRY CATCH

BEGIN TRY
    -- SQL statements executing queries and batches that might cause exceptions
END TRY
BEGIN CATCH
    -- SQL statements to handle exceptions
    BEGIN TRY
        -- Nested TRY block.
    END TRY
    BEGIN CATCH
        -- Nested CATCH block.
    END CATCH
END CATCH

For example and the demonstration, we create a table and populate it with data. let's have a simple employee table having column Name, Age, Salary, and Department. Use the below statement to create a table and populate it with data.

-- drop table employee
create table employee(
    id int identity(1,1),
    name varchar(150),
    age tinyint,
    salary decimal(10,2),
    department varchar(50)
);
set identity_insert employee on;
insert into employee (id, name, age, salary, department) values (1, 'Brittany Jeynes', 34, 75348.30, 'Support');
insert into employee (id, name, age, salary, department) values (2, 'Rita Grahamslaw', 31, 48472.11, 'Marketing');
insert into employee (id, name, age, salary, department) values (3, 'Eloise Stovin', 42, 92411.91, 'Research and Development');
insert into employee (id, name, age, salary, department) values (4, 'Angeline Blaszczyk', 30, 71372.13, 'Research and Development');
insert into employee (id, name, age, salary, department) values (5, 'Kata Yakobowitz', 42, 86797.11, 'Business Development');
insert into employee (id, name, age, salary, department) values (6, 'Minnnie Marling', 21, 101031.2, 'Human Resources');
insert into employee (id, name, age, salary, department) values (7, 'Brnaba Fenney', 42, 98107.23, 'Product Management');
insert into employee (id, name, age, salary, department) values (8, 'Verina Dyson', 42, 83881.92, 'Support');
insert into employee (id, name, age, salary, department) values (9, 'Timothee Pelz', 28, 117771.98, 'Accounting');
insert into employee (id, name, age, salary, department) values (10, 'Terrance Fomichkin', 30, 79487.77, 'Sales');
insert into employee (id, name, age, salary, department) values (11, 'Sanderson Olpin', 47, 137815.37, 'Training');
insert into employee (id, name, age, salary, department) values (12, 'Leona Melsom', 31, 142223.11, 'Engineering');
insert into employee (id, name, age, salary, department) values (13, 'Ingrid Lys', 42, 87863.52, 'Research and Development');
insert into employee (id, name, age, salary, department) values (14, 'Sophia Dungate', 25, 114351.43, 'Accounting');
insert into employee (id, name, age, salary, department) values (15, 'Estrella Kille', 31, 45259.00, 'Legal');
insert into employee (id, name, age, salary, department) values (16, 'Jonah Collecott', 28, 70264.52, 'Accounting');
insert into employee (id, name, age, salary, department) values (17, 'Urbain Squirrel', 21, 53957.32, 'Product Management');
insert into employee (id, name, age, salary, department) values (18, 'Karin Castellani', 31, 89995.29, 'Product Management');
insert into employee (id, name, age, salary, department) values (19, 'Bethena Armin', 28, 41844.10, 'Human Resources');
insert into employee (id, name, age, salary, department) values (20, 'Zonnya Trevna', 28, 56519.36, 'Training');
set identity_insert employee off;

SQL TRY CATCH Example:

For a simple try-catch example, let's try to insert a row in the employee table we created with an age column value out of the range of the tinyint data type and see how the SQL Server handles the exception.

BEGIN TRY
    insert into employee(name, age, salary, department) values ('Sana Giggs', 256, 40999, 'Sales');
END TRY
BEGIN CATCH
    SELECT  
        ERROR_MESSAGE() AS [Error Message]
        ,ERROR_LINE() AS ErrorLine
        ,ERROR_NUMBER() AS [Error Number]  
        ,ERROR_SEVERITY() AS [Error Severity]  
        ,ERROR_STATE() AS [Error State] ; 
END CATCH

As guessed correctly, we naturally will get the error while trying to insert the above statement, as the upper bound of the tinyint data type is 255 and we are trying to insert 256 in the age column, so we received an "Arithmetic overflow error for data type tinyint, value = 256." error.

SQL TRY CATCH Example in Transaction:

Similar, to the above example, we can TRY CATCH with transactions also. Let's modify the above example and include transactions in it.

BEGIN TRANSACTION;
BEGIN TRY
    update employee set age = 256 where id = 1; -- will cause same arithmetic overflow error
END TRY
BEGIN CATCH
    SELECT   
        ERROR_NUMBER() AS ErrorNumber  
        ,ERROR_SEVERITY() AS ErrorSeverity  
        ,ERROR_STATE() AS ErrorState  
        ,ERROR_PROCEDURE() AS ErrorProcedure  
        ,ERROR_LINE() AS ErrorLine  
        ,ERROR_MESSAGE() AS ErrorMessage;  
  -- @@TRANCOUNT will return a running open transaction count in the session which is not yet committed or rolled back.
    IF @@TRANCOUNT > 0  
        ROLLBACK TRANSACTION; 
END CATCH
IF @@TRANCOUNT > 0  
    COMMIT TRANSACTION;

With the execution of the above script we can see that, we received the same error. The only reason it was demonstrated, was to show an example of a transaction.

Using TRY CATCH in procedures:

To Handle exceptions in procedures lets use the below script and execute the procedure, This example is basically for showing the syntax of using try-catch in the procedure.

CREATE OR ALTER PROCEDURE [dbo].[usp_update_employee_age]
    (@input_age int, @person_id int)
    AS
    BEGIN
        BEGIN TRY
            declare @message varchar(500);
            BEGIN TRANSACTION;
                -- update age of employee
                update employee set age =  @input_age where id = @person_id;
                SELECT @message = CONVERT(VARCHAR(30), GETDATE(), 121) + ': Transaction Committed'
                print @message;
            COMMIT TRANSACTION;
        END TRY
        BEGIN CATCH
            IF @@TRANCOUNT > 0 ROLLBACK TRAN;
            SELECT @message = CONVERT(VARCHAR(30), GETDATE(), 121) + ': Transaction rolledback';
            SELECT   
                ERROR_NUMBER() AS ErrorNumber  
                ,ERROR_SEVERITY() AS ErrorSeverity  
                ,ERROR_STATE() AS ErrorState  
                ,ERROR_LINE () AS ErrorLine  
                ,ERROR_PROCEDURE() AS ErrorProcedure  
                ,ERROR_MESSAGE() AS ErrorMessage;  
            print @message;
            THROW;
        END CATCH
    END

After creating the above procedure, we can call and execute the procedure with the below statement

EXEC usp_update_employee_age 200, 1;

It will return an error with the same exception, this is a way to use try catch the transaction in the procedure. Change the value within the range of the tinyint data type and execute, it will work without any error. Now, the last important usage of try-catch which must be covered is a custom exception in SQL procedure.

Custom Exception in procedure:

Similar to custom exceptions in java and other languages, we can also have our custom exception handling inside the procedure. There are many scenarios where we need to handle certain cases and if satisfied, we need to send a custom JSON output message with an error or else a custom message with success. We can use RAISERROR or THROW for sending these custom exceptions. Note that both THROW and RAISERROR statements can be used to generate custom errors and re-throw exceptions. However, the THROW statement can re-throw the original exception that has been caught in the CATCH block and RAISERROR statement re-throws an altogether new exception and the original exception is lost. Check the script below. I am adding an extra part, wherein I validate the @input_age variable value first and raise an exception myself based on the requirement. I doing this for the demonstration and to show the syntax only, since nothing another example is coming to my mind now.

CREATE OR ALTER PROCEDURE [dbo].[usp_update_employee_age]
    (@input_age int, @person_id int, @output_json nvarchar(max) out)
    AS
    BEGIN
        BEGIN TRY
            declare @message varchar(500);
            BEGIN TRANSACTION;

                if (@input_age > 80) or (@input_age < 18)
                begin
                    -- raise an exception, since we do not want to add employee who is below 18 or above 80 years of age
                    select @output_json = N'{
                        "status" : 0,
                        "description" : "age provided for the update is not within permitted range",
                        "data": {}
                    }';
                    raiserror(@output_json, 16, 1);
                    ROLLBACK TRANSACTION;
                end
                else 
                begin
                    -- update age of employee
                    update employee set age =  @input_age where id = @person_id;
                    SELECT @message = CONVERT(VARCHAR(30), GETDATE(), 121) + ': Transaction Committed'
                    print @message;
                    set @output_json = N'{
                        "status" : 1,
                        "description" : "Employee age has been updated",
                        "data": {}
                    }';
                    RAISERROR(@output_json, 0, 1) WITH NOWAIT;
                    COMMIT TRANSACTION;
                end

        END TRY
        BEGIN CATCH
            IF @@TRANCOUNT > 0 ROLLBACK TRAN;
            SELECT @message = CONVERT(VARCHAR(30), GETDATE(), 121) + ': Transaction rolledback';
            SELECT   
                ERROR_NUMBER() AS ErrorNumber  
                ,ERROR_SEVERITY() AS ErrorSeverity  
                ,ERROR_STATE() AS ErrorState  
                ,ERROR_LINE () AS ErrorLine  
                ,ERROR_PROCEDURE() AS ErrorProcedure  
                ,ERROR_MESSAGE() AS ErrorMessage;  

            RAISERROR(@message, 0, 1) WITH NOWAIT;
            THROW;
        END CATCH
    END

After creating the above procedure, we can call and execute the procedure with the below statement

declare @output_json nvarchar(max);
EXEC usp_update_employee_age 105, 1, @output_json;

After executing the procedure, we can see the returning variable returning the response as per the case and inputs provided. The above script demonstrates the usage of custom user exceptions, and I prefer to do it this way in many cases.

The following types of errors are not handled by a CATCH block when they occur at the same level of execution as the TRY...CATCH construct:

1. Compile errors, such as syntax errors, prevent a batch from running.

2. Errors that occur during statement-level recompilation, such as object name resolution errors occur after compilation because of deferred name resolution.

3. Object name resolution errors

If any points are missing then you can add them in a comment below.

A few Things to note again are,

1. We can use the THROW function, without any parameters, but not RAISERROR.

2. Error Number for THROW function should be greater than or equal to 50000;

3. Error Severity level should be above 10 and below 19 in RAISERROR to trigger the exception and jump to catch block.

4. As may be mentioned earlier, TRY CATCH cant be used in a user-defined function.

Database Normalization is a concept & design technique we follow to efficiently organize data and design proper schema, to avoid redundant & data duplication in multiple tables. It also sets up relations between the tables, to ensure data dependency and room for future additions.

If tables are not properly normalized, then may eat up extra memory space and resources, while causing insertions, updates and deletions anomalies and complex queries.

Case

Suppose we have an LMS System for the college and no normalization design was followed while making it, then we may have many difficulties, to name a few.

In the above design of the student table, we have a few issues

Firstly, row number 2 of student_id 710003 has grouped values for an attribute(column) of program_name.

Secondly, we have added program-related attributes like program_name, and program_abbr which were unrelated to the student entity(table). We could have created a separate program table or entity and used its primary key to reference it here in the student's table and the rest program related attributes can be stored in the program table.

Thirdly, there is the repetition of data like program_name values, as many students have to opt for programs from a selected list, we can have avoided this repetition through separate program tables and maintaining the reference of keys.

While Inserting new students' data, every time the data will be repeated for program_name and abbr. And if someone had earlier made a mistake in program_name during its creation, then we will have to update each row of the said program_name in the current design.

Normalizations are guidelines and a way, for database schema & structure to be laid down.

Normally, the Third Normal Form is enough, however, you can extend it as you deem fit.

First Normal Form(1NF)

First Normal Form sets below a few guidelines for designing database table schema;

1. Define only the required and related attributes(columns) in the table.

2. Should have single-valued attributes/columns.

3. Attributes(columns) should be unique in names and the order doesn't matter.

4. Should also contain a column denoting as primary keys

Taking forward the previous table we used as an example,

Let's re-design the structure fulfilling all the guidelines.

Looking at the above table, we can see that a student with student_id 710003 has applied for 2 programs and it is inserted in comma-separated format. As we should have single-valued attributes as per the guidelines, let's restructure it and insert the second program as a new row.

The table now looks like,

By doing so, although a few values are getting repeated, values for the program_name column are now atomic for each record/row.

Using the First Normal Form, data redundancy increases, as there will be many columns with the same data in multiple rows but each row as a whole will be unique.

Second Normal Form(2NF)

The transformation of 1NF to 2NF involves the removal of partial dependencies. If a partial dependency exists, we remove the partially dependent attribute(s) from the relation by placing them in a new relation along with a copy of their determinant.

To further normalize and ensure non-repetition of data, we can break the above output table in First Normal Form into 3 different tables, as the single table above also contains information regarding program attributes that are not primary.

CREATE TABLE STUDENT(
    STUDENT_ID INT NOT NULL,
    STUDENT_NAME VARCHAR(100) NOT NULL,
    EMAIL VARCHAR(100),
    ACTIVE CHAR(1) NOT NULL DEFAULT('Y'),
    PRIMARY KEY(STUDENT_ID)
);

The above table will have the following recordset,

CREATE TABLE PROGRAM(
    PROGRAM_ID INT NOT NULL,
    PROGRAM_NAME VARCHAR(500),
    PROGRAM_ABBR VARCHAR(50),
    ACTIVE CHAR(1) NOT NULL DEFAULT('Y'),
    PRIMARY KEY(PROGRAM_ID)
);

And the program table will have the following data,

And, finally, we will create another table for storing students and their applied Programs, as we allow students to enroll in multiple programs.

CREATE TABLE STUDENT_PROGRAM(
    STUDENT_ID INT NOT NULL,
    PROGRAM_ID INT NOT NULL,
    PRIMARY KEY(STUDENT_ID, PROGRAM_ID),
    FOREIGN KEY(STUDENT_ID) REFERENCES STUDENT(STUDENT_ID),
    FOREIGN KEY(PROGRAM_ID) REFERENCES PROGRAM(PROGRAM_ID)
);

Third Normal Form(3NF)

The rule for 3NF is that the table should be in Second Normal Form and we have to ensure that non-prime attributes should not be transitively dependent on the super key(primary key). It violates the rule of the third normal form.

Again taking forward the examples we used previously, let's add state and country-related attributes to the student table.

CREATE TABLE STUDENT(
    STUDENT_ID INT NOT NULL,
    STUDENT_NAME VARCHAR(100) NOT NULL,
    EMAIL VARCHAR(100),
    STATE VARCHAR(100),
    COUNTRY VARCHAR(155),
    ACTIVE CHAR(1) NOT NULL DEFAULT('Y'),
    PRIMARY KEY(STUDENT_ID)
);

Here the State and Country in the above table are dependent on the student, so to normalize it as per Third Normalized Form, we can have another table storing the state & country list and referencing it with the id in the student's table. below is the normalized form,

CREATE TABLE STATE_COUNTRY(
    ID INT NOT NULL,
    STATE VARCHAR(100),
    COUNTRY VARCHAR (155),
    PRIMARY KEY(ID)
);

CREATE TABLE STUDENT(
    STUDENT_ID INT NOT NULL,
    STUDENT_NAME VARCHAR(100) NOT NULL,
    EMAIL VARCHAR(100),
    STATE_ID INT NOT NULL,
    ACTIVE CHAR(1) NOT NULL DEFAULT('Y'),
    PRIMARY KEY(STUDENT_ID),
    FOREIGN KEY (STATE_ID) REFERENCES STATE_COUNTRY(ID)
);

BCNF Normalization Form:

BCNF stands for Boyce-Codd Normal Form, which is an extension of 3NF. In BCNF, a table must not only contain only one type of data, have a unique identifier, and not have any transitive dependencies, but it must also not contain any repeating groups. This means that data in the table must not be repeated.

The benefit of BCNF is that it helps to improve data integrity. By eliminating repeating groups, it ensures that data is stored in the most efficient way possible. This helps to reduce data inconsistency and improve query performance.

Summary

1. Database normalization has both pros and cons. The pros include improved data integrity, reduced data redundancy, and improve query performance. The cons include increased complexity, as normalizing a database can make queries look complex due to an increase in multiple tables. It can also lead to increased storage costs, as more data may need to be stored to maintain data integrity.

2. As mentioned earlier in the start, Database normalization is an important process that helps to maintain the integrity of a database. It helps to reduce data redundancy, minimize data inconsistency, and improve query performance. It can be broken down into four major forms of normalization: 1NF, 2NF, 3NF, BCNF and more.

3. There are also pros and cons to database normalization, and it is important to understand them before implementing a database normalization strategy. Also, note that the Over-normalization of database schema also leads to an increase in complexity.

4. Denormalization is one of the last resorts for accelerating data retrieval in normalized databases when techniques like indexing are insufficient.

Finally, there are some tips and examples of database normalization that can be used to help achieve the best results.

For this purpose, I ended up creating a stored procedure using the OLE Automation procedure, which will make an HTTP request call and we can be able to perform GET and POST request methods.

OLE Automation Procedures are extended stored procedures allowing users to execute external functions to SQL Server. These can perform use cases like making HTTP calls, Reading and Writing files, Accessing File System objects, etc. The use of these extended procedures is sensitive and should be closely monitored.

The full set of OLE automation procedures are:

• sp_OACreate

• sp_OAGetProperty

• sp_OASetProperty

• sp_OAMethod

• sp_OAGetErrorInfo

• sp_OAStop

• sp_OADestroy

By default, the OLE Automation is disabled in SQL Server and has to be enabled. Below is the script to check and enable OLE Automation.

sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'Ole Automation Procedures', 1;
GO
RECONFIGURE;
GO

Let's continue forward with our demonstration.

GET Method call:

Below is the script I used in the stored procedure, for making HTTP call of GET method request type.

DECLARE @user_id int = 2; -- input parameter, to be passed with GET Request
DECLARE @URL NVARCHAR(MAX) = concat('https://reqres.in/api/users/', @user_id); -- our URL for http call
print '@URL -> '+ @URL;
Declare @Object as Int; -- declare object
Declare @ResponseText as Varchar(8000);

Exec sp_OACreate 'MSXML2.XMLHTTP', @Object OUT; -- creating OLE object and assigning it to variable @Object
-- print @Object; -- 16711422

-- passing the @Object created above, with our http call and handling the response with help of sp_OAMethod 
Exec sp_OAMethod @Object, 'open', NULL, 'get',
       @URL,
       'False'
Exec sp_OAMethod @Object, 'send'
Exec sp_OAMethod @Object, 'responseText', @ResponseText OUTPUT
IF((Select @ResponseText) <> '')
BEGIN
     DECLARE @json NVARCHAR(MAX) = (Select @ResponseText)
     print(@json); -- printing the json output

     select 
        rs.id, rs.email, rs.first_name, rs.last_name, rs.avatar
     from openjson(@json, '$.data') with  -- only parsing data json array object of the responseText
     (
        id int '$.id',
        email varchar(200) '$.email',
        first_name varchar(200) '$.first_name',
        last_name varchar(200) '$.last_name',
        avatar varchar(255) '$.avatar'
     ) rs; -- select query to further process the data.
END
ELSE
BEGIN
     DECLARE @ErroMsg NVARCHAR(30) = 'No data found.';
     Print @ErroMsg;
END
Exec sp_OADestroy @Object; -- destory the object when not needed anymore. If not called, the OLE object will be destroyed automatically when the batch execution is completed.

POST Method call:

Below is the script, which can be used in the stored procedure, for making HTTP calls of POST method request type.

DECLARE @URL NVARCHAR(MAX) = 'https://reqres.in/api/users/'; -- our URL for post request
DECLARE @Object AS INT; -- object declaration
DECLARE @ResponseText AS VARCHAR(8000);
DECLARE @Body AS VARCHAR(8000) =

'{
    "name": "kiran sabne",
    "job": "leader"
}';
EXEC sp_OACreate 'MSXML2.XMLHTTP', @Object OUT; -- creating OLE object and assigning it to variable @Object

-- passing the @Object created above, with our http call and handling the response with help of sp_OAMethod 
EXEC sp_OAMethod @Object, 'open', NULL, 'post',
                 @URL,
                 'false'
EXEC sp_OAMethod @Object, 'setRequestHeader', null, 'Content-Type', 'application/json'
EXEC sp_OAMethod @Object, 'send', null, @body
EXEC sp_OAMethod @Object, 'responseText', @ResponseText OUTPUT
-- print 'responseText -> ' + @ResponseText;
IF CHARINDEX('false',(SELECT @ResponseText)) > 0
BEGIN
 SELECT @ResponseText As 'Message'
END
ELSE
BEGIN
 SELECT @ResponseText As 'New User Added Detail'
END
EXEC sp_OADestroy @Object

Summary

As mentioned earlier, the usage of OLE Procedures is sensitive and must be strictly monitored. Similarly, we can also perform File Object modifications with the OLE procedures. The other obvious and preferred alternatives are creating another application server for doing work and that application having access to the database.

drop table if exists orders_demo;
create table orders_demo(
    order_date datetime2 not null,
    order_amount numeric(10,2) not null default(0.0)
);

insert into orders_demo (order_date, order_amount) values ('2021-08-29 06:24:36', 96645.23);
insert into orders_demo (order_date, order_amount) values ('2021-08-17 12:57:42', 2799.66);
insert into orders_demo (order_date, order_amount) values ('2021-12-08 01:21:10', 36855.52);
insert into orders_demo (order_date, order_amount) values ('2021-09-04 02:58:29', 12745.25);
insert into orders_demo (order_date, order_amount) values ('2021-06-02 05:52:27', 19305.53);
insert into orders_demo (order_date, order_amount) values ('2021-08-01 21:35:38', 44003.67);
insert into orders_demo (order_date, order_amount) values ('2021-06-29 07:31:51', 9620.3);
insert into orders_demo (order_date, order_amount) values ('2021-05-16 13:41:58', 83331.23);
insert into orders_demo (order_date, order_amount) values ('2021-05-14 10:34:52', 52047.06);
insert into orders_demo (order_date, order_amount) values ('2021-10-28 10:52:08', 87767.65);
insert into orders_demo (order_date, order_amount) values ('2021-09-07 15:30:38', 79037.05);
insert into orders_demo (order_date, order_amount) values ('2021-06-26 17:09:48', 45204.9);
insert into orders_demo (order_date, order_amount) values ('2021-05-23 17:32:55', 1529.24);
insert into orders_demo (order_date, order_amount) values ('2021-08-07 15:40:40', 40596.98);
insert into orders_demo (order_date, order_amount) values ('2021-07-18 07:45:15', 72129.59);
insert into orders_demo (order_date, order_amount) values ('2021-10-29 16:29:39', 5195.02);
insert into orders_demo (order_date, order_amount) values ('2021-10-14 10:21:15', 21266.39);
insert into orders_demo (order_date, order_amount) values ('2021-09-13 06:21:35', 43578.89);
insert into orders_demo (order_date, order_amount) values ('2021-06-25 04:57:38', 5705.68);
insert into orders_demo (order_date, order_amount) values ('2021-10-04 17:41:11', 48334.64);
insert into orders_demo (order_date, order_amount) values ('2021-09-24 08:54:28', 90635.05);
insert into orders_demo (order_date, order_amount) values ('2021-11-01 07:47:06', 9952.95);
insert into orders_demo (order_date, order_amount) values ('2021-05-17 09:05:54', 82753.71);
insert into orders_demo (order_date, order_amount) values ('2021-06-21 21:11:44', 44711.68);
insert into orders_demo (order_date, order_amount) values ('2021-06-20 09:02:21', 95524.26);
insert into orders_demo (order_date, order_amount) values ('2021-08-11 17:24:09', 51805.88);
insert into orders_demo (order_date, order_amount) values ('2021-07-08 02:23:24', 56710.13);
insert into orders_demo (order_date, order_amount) values ('2021-12-14 14:15:48', 39827.63);
insert into orders_demo (order_date, order_amount) values ('2021-09-08 09:04:30', 70854.91);
insert into orders_demo (order_date, order_amount) values ('2021-07-23 01:42:31', 36303.79);

-- select * from orders_demo

PIVOT

Looking at the dataset we can see that it has a date range of a few months. Now suppose we want to arrange and transpose the dataset in such a way that, we want to show average order amounts for each month in the data set as month-wise columns.

select 
    'average_order_amount' as avg_order_amounts,
    [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12]
from 
(
    select month(order_date) as month_num, order_amount
        from 
    orders_demo
) as s
pivot 
(
    avg(order_amount)
    for month_num in ([1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12])
) as avg_pivot_month

The above SQL snippet will return a row with columns for all 12 months and the average order total as its value. The first 4 months will be having NULL value as we don't have a dataset for those date ranges.

UNPIVOT

As mentioned earlier in the introduction part, UNPIVOT operator transposes multiple columns to a single column with multiple rows of records and returns as a dataset. In the earlier part, we used the PIVOT operator on the orders_demo table and returned a row containing a name column and 12 other columns for each month of the year. Now suppose, if we want the result set such that, it will have only two columns, one named month_num depicting the month number and the other average_order_amount depicting the average of all the order amount for the month from the orders_demo table.

To transpose the dataset in such a way, we will use the above pivot query as a source and return the values of the multiple columns of the source result set in a single column as the average order amount for the month.

select
    month_num,
    average_amounts
from
(
    select 
        'average_order_amount' as avg_order_amounts,
        [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12]
    from 
    (
        select month(order_date) as month_num, order_amount
            from 
        orders_demo
    ) as s
    pivot 
    (
        avg(order_amount)
        for month_num in ([1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12])
    ) as avg_pivot_month
) source_set 
unpivot
(
    average_amounts for month_num in ([1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12])
) as avg_unpivot;

In the above SQL snippet, we first used the PIVOT operator to return a row containing a name for the average_name column along with 12 other columns containing the values of the average order amount for each month, the result set is named avg_pivot_month. Later using that result set with UNPIVOT operator we transposed those 12 multiple columns of the month to 1 column named month_num and its values in the average_amounts column. And remember that since there is no record for the first 4 months in our table set, we will not get those months in the month_num column. Note that UNPIVOT operator doesn't split up the aggregated results, hence UNPIVOT operator is not exactly the opposite of the PIVOT operator.

In the next post regarding PIVOT and UNPIVOT, we will be trying to implement a dynamic SQL PIVOT and UNPIVOT script/function with an example dataset.

CREATE DATABASE DemoDB;

use DemoDB;

CREATE TABLE products(
   id int primary key identity(1,1),
   item_name varchar(20) not null,
   item_description varchar(40),
   unit_price decimal(8,2)
);

SET IDENTITY_INSERT [dbo].[products] ON 
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (1, N'Compactor', NULL, CAST(7145.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (2, N'Crawler', NULL, CAST(9549.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (3, N'Excavator', NULL, CAST(39549.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (4, N'Scraper', NULL, CAST(4883.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (5, N'Grader', NULL, CAST(9549.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (6, N'Trencher', NULL, CAST(7347.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (7, N'Backhoe', NULL, CAST(1275.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (8, N'Dragline', NULL, CAST(9549.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (9, N'Dump Truck', NULL, CAST(25241.00 AS Decimal(8, 2)))
GO
INSERT [dbo].[products] ([id], [item_name], [item_description], [unit_price]) VALUES (10, N'Item 1', NULL, CAST(500.00 AS Decimal(8, 2)))
GO
SET IDENTITY_INSERT [dbo].[products] OFF
GO

SELECT * FROM products;

Enabling Change Tracking

To enable CT on the table, we have to enable it on the database level first. Since we have created a demo database for the illustration with the table and its data, let's enable Change Tracking for our DemoDB database.

use DemoDB;

ALTER DATABASE DemoDB
SET CHANGE_TRACKING = ON
(CHANGE_RETENTION = 2 DAYS, AUTO_CLEANUP = ON);

As the alter statement above enables Change_Tracking on Database Level, we need to mention retention value and CleanUp value. Change_retention field requires value and unit, so the data can be retained for the mentioned period in the internal on-disk tables and the Auto_cleanup field takes ON or OFF as value, for enabling or disabling the auto cleanup process. If auto cleanup is enabled then it will delete the internal on-disk tables data after the mentioned retention period automatically. After enabling CT on the Database level, we also have to enable it for each table we want to track.

USE DemoDB
GO

ALTER TABLE products
ENABLE CHANGE_TRACKING
WITH (TRACK_COLUMNS_UPDATED = ON);

With the alter statement above, we enabled the change tracking for the table we wanted. We could also do the enabling of both, the database level and table level through the management studio, by visiting database properties and table properties respectively. Now let's see how Change Tracking works with each DML event.

Before moving forward, we need to know one thing, it is about the version number counter, and is incremented automatically with any DML operation on the tracked table. To check the current version of the tracked table, use the below query

SELECT CHANGE_TRACKING_CURRENT_VERSION();

DML Events making Changes

Insert Operation

To see the working of Change Tracking in the case of an Insert operation, let's insert 2 new records in our product table which is a tracked table, and see how we can track the new insertions

INSERT INTO products(item_name, unit_price) VALUES
    ('Item 1', 322.45),
    ('Item 2', 137.90);

We can utilize the Change Tracking functions and CHANGETABLE system function, to access the data saved in the internal table after the INSERT statement executed above. Each DML event causing data modifications increments the version number counter. The function returns all the changes made on the tracked table with the version counter. Query the select statement given below to view the modifications in the tracked table.

SELECT * FROM CHANGETABLE 
(CHANGES products,0) as CT ORDER BY SYS_CHANGE_VERSION;

image: In the image, we can see the version counter, type of DML Event, which is I in our case as we did inserts along with the primary key column as value for the newly inserted rows. We also see that SYS_CHANGE_COLUMNS, SYS_CHANGE_CONTEXT column currently has a NULL value. SYS_CHANGE_COLUMNS returns the list of columns which was modified since the last version counter and are of binary data type. Also, note that Inserts and Deletes will always have an SYS_CHANGE_COLUMNS value "null", a whole row is affected by an insert or a delete. SYS_CHANGE_CONTEXT is optional and is used to specify the context changes using WITH clause while performing DML changes.

Update Operation

Like the insert operation, let's perform the update operation, by updating unit_price to 500.00 for product id 10 and also two updates for product id 11 in the products table. Before proceeding to execute the update statement, check the current version number.

UPDATE products SET unit_price = 500.00 WHERE id = 10;
UPDATE products SET unit_price = 500.00 WHERE id = 11;
UPDATE products SET unit_price = 520.00 WHERE id = 11;

Now let's see the effect of the above update statement in our internal table, with the same select statement executed above.

SELECT * FROM CHANGETABLE 
(CHANGES products, 1) as CT ORDER BY SYS_CHANGE_VERSION;

image:

In the snapshot above, we can see that SYS_CHANGE_VERSION which is maintaining the counter incremented for the id 10 along with U Char for the SYS_CHANGE_OPERATION column denoting that the last operation on the row was an update statement, and SYS_CHANGE_COLUMNS contains the binary value for the columns modified. Notice that the above select statement is a little different than the previous select statement we used in the Insert part. Along with the table name in the CHANGES sub-clause we also provided 1 as a literal value which was my current version counter before the update operation. Mentioning it in the select query from the changetable, we get the record of the modification made after the last version mentioned. Even though product id 11 has been updated twice, we only can get the record of the most recent modification made to the row. It doesn't provide the history of data changes made. Now let's see how a delete operation is recorded in the Changetable internal table.

Delete

Now, let's delete the product having id 11 from the products table, by executing the below delete statement. Before that, you can check the current version, which we use to query changetable later. My current counter value is 4.

DELETE FROM products where id = 11;

Now let's query the changetable and see the records reflecting our delete operation

SELECT * FROM CHANGETABLE 
(CHANGES products, 4) as CT ORDER BY SYS_CHANGE_VERSION;

image: In the above select query to fetch the records from the changetable for the table products, we passed the literal value 4 after the comma in the CHANGES sub-clause, it was my version counter value before the delete statement was executed. And we can see the version counter has incremented and the SYS_CHANGE_OPERATION value is Character D denoting the Delete operation. SYS_CHANGE_COLUMNS depicting the list of the columns changed will be null, as it was a delete operation. Now, let's run the same select query on the changetable for the products table, but this time we will provide literal value 1 in place of 4 for the CHANGES clause within the brackets. This will return the last modifications made in the products table after our first insertion for the records.

SELECT * FROM CHANGETABLE 
(CHANGES products, 1) as CT ORDER BY SYS_CHANGE_VERSION;

image: In the snapshot of the output above, we can infer that the changetable only records the last modification made for the particular record row. Earlier in the Update part, we perform an update operation twice on product id 11 still it only shows the record of the last operation for product id 11.

Disable Change_Tracking

To disable the Change_Tracking (CT) function, we need to first disable tracking for the tracked tables. After disabling tracking on the table, we can disable it for the database with the T-SQL code below

ALTER TABLE products
DISABLE CHANGE_TRACKING
GO

ALTER DATABASE DemoDB
SET CHANGE_TRACKING = OFF
GO

The Change_Tracking function has limited usage and it's mainly the best use case where we want to track the data changes on the table, but we have no requirement for the pre and post-data modifications, or we only want to know if the row is modified or not. One such use case is implementing a custom cache layer for the application where we can synchronize our cache database for the application or reinitialize it as the case may be. In the next post for Change Tracking, we will be looking at a few Change Tracking Functions.

• Read Uncommitted

• Read Committed (The default)

• Repeatable Read

• Serializable

• Snapshot Isolation and its example are already covered in another post links to them are at the end of this post. Before moving forward to Isolation levels one by one, let's create a table with some mock data for some illustrations later.

drop table if exists product_listing;
create table product_listing(
    id int identity(1,1),
    item_name varchar(50) not null,
    description varchar(100),
    unit_price decimal(10,2)
);
insert into product_listing(item_name, unit_price) values ('Compactor', 7145.00);
insert into product_listing(item_name, unit_price) values ('Crawler', 9549.00);
insert into product_listing(item_name, unit_price) values ('Excavator', 39549.00);
insert into product_listing(item_name, unit_price) values ('Scraper', 4883.00);
insert into product_listing(item_name, unit_price) values ('Grader', 9549.00);
insert into product_listing(item_name, unit_price) values ('Trencher', 7347.00);
insert into product_listing(item_name, unit_price) values ('Backhoe', 1275.00);
insert into product_listing(item_name, unit_price) values ('Dragline', 9549.00);
insert into product_listing(item_name, unit_price) values ('Dump Truck', 25241.00);

Now, let's start.

1. Read Uncommitted; Read Uncommitted is the lowest level of isolation, it enables the Transaction to read uncommitted data modifications made by other transactions concurrently on the same resources. The transaction can read the row values even when locks are applied on that row by other transactions and that other transaction has not committed or rollbacked either yet. This case is also known as Dirty Reads. Since the other transaction modifying the rows hasn't yet made a commit or rollback, we may see unexpected results. However, there are cases where we might be required to read uncommitted data.

Run both the below transactions in two separate query editor tabs. In our example, let's try to update the unit_price of the product having id as 1 in the product_listing table. And let's try to read that product id row in another transaction.

Begin Transaction

Update product_listing set unit_price = 10000.00 where id = 1;

waitfor delay '00:00:10';

Rollback Transaction ;

Now in another window tab, try to read the product_listing table for that same id, i.e. 1

set transaction isolation level read uncommitted;

select * from product_listing where id = 1; 

waitfor delay '00:00:10';

In the first transaction, we updated the unit_price of product id 1 of the product_listing table and simulated a delay so we can read the value of product id 1 in the second transaction. On the other query editor screen when executing the second transaction we can see that even though the update statement in the first transaction has an exclusive lock for the row, it's readable by the select statements in the second transaction. The first select statement in the second transaction returns the updated unit_price of the product_id 1 and then waits for the first transaction to roll back, to read again for the same product and find out that the unit_price is changed again and it reverted to the old value. Simply saying the second transaction returned the value that was there at that time. Another syntax for reading uncommitted data is to use the NOLOCK table hint, which then read uncommitted data even if the isolation level is not mentioned in the transaction.

select * from product_listing where id = 1 with(NOLOCK);

1. Read Committed: This is the default isolation level and it guarantees that the return of any data read is the committed value at the time of reading. In simpler terms, the read operation will wait, for any pending transaction on the same table to complete and will return the committed value. The shared lock request on the data will wait for the exclusive lock to commit if any pending in other transactions for the same data. Let's simulate it with example table data.

BEGIN TRANSACTION

Update product_listing set unit_price = 10000.00 where id = 1;

waitfor delay '00:00:10';

ROLLBACK TRANSACTION;

In another query editor window tab, we will read the data for the same id.

select * from product_listing where id = 1;

Executing both transactions in a different window, we can see that the second transaction, even though it's only a read operation, it waits for the first transaction to complete (i.e. either commit or rollback) and on completion of the first transaction, the second transaction can read the data. We have to remember that the read operation will issue shared lock requests against data we want to read, and it will wait if another transaction already has an exclusive lock on that data. If both the transactions working on the same data want only a shared lock, then there won't be any execution delay for the transaction, as they don't need to wait for the other transaction to complete. The transaction holds a read or write lock on the current row, and thus prevents other transactions from reading, updating, or deleting it.

1. Repeatable Read IN THIS ISOLATION LEVEL, every time we read the same record set it returns the same value till the end of the transaction. It blocks the write operation trying to update the same record set we are reading, by holding a shared lock till the transaction is complete. Thus we see the same results within the transaction. It is to note here that, Repeatable Read blocks update and delete operation for the same recordset which obtained the shared lock, but it doesn't block insert operation. If the new records are satisfying the condition of the select statement, it will also return those newly added records along with the previous record set. Let's simulate both cases.

begin transaction;
set transaction isolation level repeatable read;

select * from product_listing where unit_price < 2000; 

waitfor delay '00:00:10';

select * from product_listing where unit_price < 2000;

commit transaction;

In the second transaction in the new window, we will try to update the unit_price of product id 7 to 1900.

-- transaction 2
Update product_listing set unit_price = 1900 where id = 7;

But if we read the data after all the transactions are completed, we can see that unit_price of the product id 7 is updated to 1900.00 This shows us how Repeatable Read Isolation prevents data modification till the end of the transaction. Let's also simulate the second case, to see how the insert operation will work with the isolation level.

begin transaction;
set transaction isolation level repeatable read;

select * from product_listing where unit_price < 2000;

waitfor delay '00:00:10'; 

select * from product_listing where unit_price < 2000; 

commit transaction;

In the second transaction, another window, we will insert another new product having unit_price below 2000.

insert into product_listing(item_name, unit_price) values ('new item 1', 1500.00);

As we can see on both the transaction execution, even though the first transaction has obtained a shared lock on the same read condition, still the second transaction returns immediately without waiting for the first transaction execution to finish. Now when we perform the read operation for the second time, in transaction one after transaction two completes its insertion, we get two rows returned, old records along with new records as it satisfies the WHERE conditions. Do remember that, Repeatable Read doesn't block insert operation, thus leading to phantom read in some cases.

1. Serializable This is similar to the Repeatable Read isolation level with an extra added layer that prevents insert operations also. This eradicates the problem of phantom reads. Serializable Isolation level uses range locks on the record set, so it can't be modified, inserted, or deleted till the end of the transaction. Let's simulate the previous example to see if the insert operations are blocked till the end of the transaction.

begin transaction;
set transaction isolation level serializable;

select * from product_listing where unit_price < 2000;

waitfor delay '00:00:10'; 

select * from product_listing where unit_price < 2000;

commit transaction;

In the second transaction, another window, we will insert another new product having unit_price below 2000.

insert into product_listing(item_name, unit_price) values ('new item 2', 1700.00);

With both transaction execution, we can see that the second transaction which has an insert operation doesn't return immediately like in Repeatable Read Isolation. The second transaction waits till the first transaction is complete and on completion of the first transaction, the second transaction performs the insert operation for adding a new record. This demonstrates the blocking of all DML operations from another transaction, till the current transaction is completed under Serializable Isolation. It is to be noted that, if the table has an index on it, then the records are locked based on those index ranges used in the WHERE clause, otherwise it locks the complete table for the transaction.

This completes our small post of transaction isolation level. For the Snapshot Isolation, kindly look at the below-given links.

• SQL Server Concurrency with Row Versioning

DROP TABLE IF EXISTS [dbo].sales;
CREATE TABLE [dbo].sales(
    sales_id int primary key,
    customer_id int not null,
    sales_person_id int not null,
    order_date date not null,
    total_amount_including_taxes decimal(10,2) not null default(0),
    tax_amount decimal(8,2) not null default(0),
    delivery_date date not null,
    extra_instruction varchar(500),
    updated_by int not null,
    updated_timestamp datetime2(0) not null default current_timestamp
);

INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (1, 4, 2, '2021-09-27', 1704.69, 49.81, '2021-09-29', 'in tempor turpis nec euismod scelerisque quam turpis adipiscing lorem vitae mattis nibh ligula nec sem duis aliquam convallis nunc', 2, '2021-09-27');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (2, 3, 3, '2021-10-25', 3314.02, 99.97, '2021-10-27', 'consectetuer eget rutrum at lorem integer tincidunt ante vel ipsum praesent blandit lacinia erat vestibulum sed magna', 3, '2021-10-25');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (3, 5, 2, '2021-09-25', 9147.42, 54.77, '2021-09-30', 'nisl ut volutpat sapien arcu sed augue aliquam erat volutpat in congue etiam justo etiam pretium iaculis justo', 2, '2021-09-25');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (4, 9, 2, '2022-01-27', 5173.06, 26.01, '2022-01-30', 'ipsum dolor sit amet consectetuer adipiscing elit proin risus praesent', 2, '2022-01-27');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) values (5, 1, 2, '2022-04-27', 8838.68, 33.07, '2022-04-30', 'ac tellus semper interdum mauris ullamcorper purus sit amet nulla quisque arcu libero rutrum ac lobortis', 2, '2022-04-27');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (6, 8, 2, '2021-05-02', 5959.05, 61.41, '2021-05-27', 'ut ultrices vel augue vestibulum ante ipsum primis in faucibus orci luctus et ultrices', 2, '2021-05-02');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (7, 2, 3, '2022-04-15', 1521.54, 78.27, '2021-04-21', 'vel lectus in quam fringilla rhoncus mauris enim leo rhoncus', 3,  '2022-04-15');
insert into sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (8, 9, 1, '2021-08-25', 6301.25, 66.37, '2021-08-30', 'ut ultrices vel augue vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae', 1, '2021-08-25');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (9, 5, 1, '2021-09-29', 6894.79, 15.74, '2021-10-29', 'pede ullamcorper augue a suscipit nulla elit ac nulla sed vel', 1, '2021-09-29');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (10, 5, 4, '2021-12-08', 8310.35, 35.54, '2021-12-31', 'mi integer ac neque duis bibendum morbi non quam nec', 4,  '2021-12-08');
 INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (11, 8, 2, '2021-10-13', 5815.76, 95.91, '2021-10-20', 'non mi integer ac neque duis bibendum morbi non quam nec dui luctus rutrum', 2, '2021-10-13');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (12, 3, 3, '2022-04-24', 1860.23, 65.36, '2022-04-24', 'odio porttitor id consequat in consequat ut nulla sed accumsan felis ut at dolor quis odio consequat varius integer', 3, '2022-04-24');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (13, 2, 1, '2022-01-22', 8716.47, 31.77, '2022-01-28', 'nec condimentum neque sapien placerat ante nulla justo aliquam quis turpis eget', 1, '2022-01-22');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (14, 6, 2, '2022-03-26', 9954.56, 29.54, '2022-03-27', 'in purus eu magna vulputate luctus cum sociis natoque penatibus et magnis dis parturient montes nascetur ridiculus', 2,'2022-03-26');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (15, 8, 1, '2021-07-31', 4671.74, 43.64, '2021-08-16', 'nisl nunc rhoncus dui vel sem sed sagittis nam congue risus semper porta', 1, '2021-07-31');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (16, 1, 2, '2022-02-25', 3511.53, 74.65, '2022-02-25', 'pede ac diam cras pellentesque volutpat dui maecenas tristique est et tempus semper est quam pharetra magna ac consequat', 2, '2022-02-25');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (17, 8, 1, '2021-10-01', 8507.02, 59.93, '2021-10-14', 'amet erat nulla tempus vivamus in felis eu sapien cursus vestibulum proin eu mi nulla ac enim in tempor', 1, '2021-10-01');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (18, 6, 4, '2022-01-08', 8008.41, 69.92, '2022-01-28', 'venenatis lacinia aenean sit amet justo morbi ut odio cras mi pede malesuada in imperdiet et commodo vulputate', 4, '2022-01-08');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (19, 1, 2, '2022-02-11', 1291.84, 19.51, '2022-02-15', 'eget elit sodales scelerisque mauris sit amet eros suspendisse accumsan tortor quis turpis sed ante vivamus', 2, '2022-02-11');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (20, 8, 4, '2022-03-25', 2948.43, 22.37, '2022-03-27', 'curae nulla dapibus dolor vel est donec odio justo sollicitudin', 4, '2022-03-25');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (21, 5, 4, '2021-06-12', 5466.84, 35.58, '2021-06-30', 'in faucibus orci luctus et ultrices posuere cubilia curae mauris viverra', 4, '2021-06-12');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (22, 9, 3, '2021-11-30', 8762.52, 22.53, '2021-12-30', 'ultrices vel augue vestibulum ante ipsum primis in faucibus orci luctus et', 3, '2021-06-12');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (23, 5, 1, '2021-05-18', 7894.89, 51.04, '2021-05-18', 'id pretium iaculis diam erat fermentum justo nec condimentum neque', 1, '2021-05-18');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (24, 4, 3, '2022-01-20', 9051.32, 22.23, '2022-01-20', 'nibh in hac habitasse platea dictumst aliquam augue quam sollicitudin', 3, '2022-01-20');
INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) values (25, 7, 3, '2022-03-20', 2537.84, 36.42, '2022-03-30', 'velit nec nisi vulputate nonummy maecenas tincidunt lacus at velit', 3, '2022-03-20');

DROP TABLE IF EXISTS [dbo].sales_log;
CREATE TABLE [dbo].sales_log(
    log_id int identity(1,1),
    sales_id int not null,
    customer_id_old int,
    customer_id_new int,
    sales_person_id_old int,
    sales_person_id_new int,
    order_date_old date,
    order_date_new date,
    total_amount_including_taxes_old decimal(10,2),
    total_amount_including_taxes_new decimal(10,2),
    tax_amount_old decimal(8,2),
    tax_amount_new decimal(5,2),
    delivery_date_old date,
    delivery_date_new date,
    extra_instruction_old varchar(500),
    extra_instruction_new varchar(500),
    updated_by_old int,
    updated_by_new int,
    updated_timestamp_old datetime2(0),
    updated_timestamp_new datetime2(0),
    action_done varchar(6) not null,
    action_time datetime2(0) not null,
    username varchar(128) null
);

The sales_log table inserts both old and new values of the columns, for every write operation in the sales table. It helps in tracking data changes and the columns action_done, action_time, and username in the sales_log table with DML operation which caused the change, timestamp, and database user. Normally we create a different user in the database for the application, so it's necessary to know the database user. Now let's create a trigger that tracks data changes in the sales table.

CREATE TRIGGER TR_audit_sales_table
    ON [dbo].sales
    AFTER INSERT, UPDATE, DELETE
AS
BEGIN
    SET NOCOUNT ON;
    INSERT INTO [dbo].sales_log 
        (sales_id,
        customer_id_old,
        customer_id_new,
        sales_person_id_old,
        sales_person_id_new,
        order_date_old,
        order_date_new,
        total_amount_including_taxes_old,
        total_amount_including_taxes_new,
        tax_amount_old,
        tax_amount_new,
        delivery_date_old,
        delivery_date_new,
        extra_instruction_old,
        extra_instruction_new,
        updated_by_old,
        updated_by_new,
        updated_timestamp_old,
        updated_timestamp_new,
        action_done,
        action_time,
        username
        )
    SELECT
        ISNULL(i.sales_id, d.sales_id) AS sales_id,
        d.customer_id,
        i.customer_id,
        d.sales_person_id,
        i.sales_person_id,
        d.order_date,
        i.order_date,
        d.total_amount_including_taxes,
        i.total_amount_including_taxes,
        d.tax_amount,
        i.tax_amount,
        d.delivery_date,
        i.delivery_date,
        d.extra_instruction,
        i.extra_instruction,
        d.updated_by,
        i.updated_by,
        d.updated_timestamp,
        i.updated_timestamp,
        CASE
            WHEN i.sales_id IS NULL THEN 'DELETE'
            WHEN d.sales_id IS NULL THEN 'INSERT'
            ELSE 'UPDATE'
        END AS action_done,
        SYSUTCDATETIME(),
        SUSER_SNAME()
    FROM inserted i
    FULL JOIN deleted d
    ON i.sales_id = d.sales_id;
END

The above trigger code inserts new rows in the sales_log table for every write operation on the sales table and it can be further modified as per the requirements.

With the trigger created and executed, now we will test the working with some write operations in the sales table. Let us insert a new sale record in the sales table and see the information captured in the sales_log table.

INSERT INTO sales (sales_id, customer_id, sales_person_id, order_date, total_amount_including_taxes, tax_amount, delivery_date, extra_instruction, updated_by, updated_timestamp) VALUES (26, 7, 3, '2022-03-22', 25337.84, 361.42, '2022-05-01', 'velit nec nisi vulputate nonummy maecenas tincidunt lacus at velit', 3, '2022-03-20');

With a new sale record added having sales_id as 26, we can see the captured data in the sales_log table with the action type and database username which made those changes. A few of the columns in the sales_log table will be NULL since old data does not exist in the insert operation. Now let's update the same sale record updating the sales_person_id, total_amount_including_taxes and tax_amount.

UPDATE sales SET
    sales_person_id = 2,
    total_amount_including_taxes = 30000.00,
    tax_amount = 300.00
WHERE sales_id = 26;

When we check the log of the update operation in the sales_log table we can see that the trigger has captured and inserted both old data values and new data values for all columns and also see that the action_done is set to 'UPDATE'; Now let's delete a sale record from the sales table and see its insert in the sales_log table.

DELETE FROM sales WHERE sales_id = 26;

Now when we inspect the sales_log table for the above delete operation, we only see old values for the columns and NULL for the corresponding columns holding new values. This is how we can use the trigger for auditing and implementing data changes capture in SQL Server.

Related Post: Understanding Triggers in SQL Server

The Trigger definition can either be of type INSTEAD OF or AFTER or FOR. When we perform a write operation on a table, the INSTEAD OF trigger replaces the operation and executes trigger content instead and in the case of AFTER trigger the trigger gets fired once the write operation is completed. Do Note that if the AFTER Trigger fails, the original write operation also fails.

The automated execution of code on every change event is the main plus point of the trigger and also if the trigger code fails, the event which fired the trigger will fail. A few cases where triggers serve their purpose are tracking database changes including schema or object changes, writing table logs, and validation checks and also they can implement custom validation functions, enforcing other business rules, and exert control when performing an insert, update, or delete operations and decide what to do with data. Before moving forward let's create a simple trigger example.

Basic Trigger Example

Consider a case, wherein we want to log all update operations on the Users table and record those operations in the users_log table for our future audit purpose. Let's start by creating a Users table and inserting some data in it.

DROP TABLE IF EXISTS users;
CREATE TABLE users(
    id INT IDENTITY(1,1) PRIMARY KEY,
    first_name VARCHAR(30) not null,
    last_name VARCHAR(35),
    date_of_birth DATE,
    user_type_id INT
);

INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Lamont', 'Lantiffe', 3, '1996-10-03');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Xenia', 'Donovin', 4, '1996-10-19');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Eleen', 'Joriot', 4, '1997-01-19');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('De', 'Alpe', 1, '1996-12-04');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Skipton', 'Myrick', 4, '1996-03-23');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Kariotta', 'Sapsforde', 1, '1996-12-10');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Darill', 'De Moreno', 1, '1996-09-22');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Sayre', 'Critch', 3, '1996-07-29');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Chrissy', 'Vedekhov', 3, '1996-08-27');
INSERT INTO users (first_name, last_name, user_type_id, date_of_birth) VALUES ('Dalis', 'Dutnall', 1, '1996-08-20');

DROP TABLE IF EXISTS users_log ;
CREATE TABLE users_log (
    users_log_id int identity(1,1) primary key,
    users_id int,
    first_name_old varchar(30),
    first_name_new varchar(30),
    last_name_old varchar(35),
    last_name_new varchar(35),
    date_of_birth_old date,
    date_of_birth_new date,
    user_type_id_old int,
    user_type_id_new int,
    action_type varchar(10),
    action_time datetime2 not null,
    user_name varchar(128)
);

CREATE TRIGGER dbo.TR_U_Users_Audit 
on dbo.Users
AFTER UPDATE
AS
BEGIN
    SET NOCOUNT ON;

    INSERT INTO dbo.users_log
        (users_id, first_name_old, first_name_new, last_name_old, last_name_new, date_of_birth_old, date_of_birth_new, user_type_id_old, user_type_id_new, action_type, action_time, user_name)
    SELECT i.id, d.first_name as first_name_old, i.first_name as first_name_new, d.last_name as last_name_old, i.last_name as last_name_new, d.date_of_birth as date_of_birth_old, i.date_of_birth as date_of_birth_new, d.user_type_id as user_type_id_old, i.user_type_id as user_type_id_new,
    'Update', SYSUTCDATETIME() action_time,
        SUSER_SNAME() AS user_name 
    FROM Inserted i
    FULL JOIN Deleted d
        ON i.id = d.id;
END
GO

The above-created triggers' sole purpose is to insert records in Users_Audit Table, one for each UPDATE operation modifying the data in the Users table. It's simple and we can further modify it if we want to track any other details regarding it.

SELECT * FROM users;

UPDATE users SET first_name = 'Leaim', last_name = 'Lantiffe', date_of_birth = '1994-01-13' WHERE id = 1;

SELECT * FROM users_log;

With the above update operation, the old and the new data of the record are captured in the Users_Log Table. The Users_log Table contains both old and new data for the updated user_id 1. We have joined Inserted and Deleted pseudo tables in the triggers to get both the old and new data of the record. Let's understand INSERTED and DELETED pseudo tables more, so we can use them to write better triggers.

INSERTED and DELETED table.

INSERTED and DELETED are special pseudo tables, available for the write operations. This pseudo table is available during the execution of DML events. Insert operation will have INSERTED table, Delete operation will have DELETED table, and Update operation will have both INSERTED and DELETED table. In the case of Insert operations, the INSERTED table will contain new values for each column in a table and the DELETED table will have no data. Similarly, for the Delete operation, the DELETED table will contain old values for each column in a table and the INSERTED table will have no data. And for the Update operation, the INSERTED table will contain new data for each column in a table and the DELETED table will contain old data. This construct makes it easier to identify the type of writing operation being conducted on a table. If both the INSERTED and DELETED table has data, then it was an Update Operation, if only the DELETED table didn't have any data then it's an Insert operation and if the INSERTED table didn't have any data then it's a Delete operation. This helps us in understanding why we had joined INSERTED and DELETED tables in the trigger written above for logging all update operations on the Users Table. Now we will see more examples for each DML operation with the 'AFTER' and 'INSTEAD OF' triggers.

For further examples, let's create two tables one containing the records of Invoices and the other containing the invoice line items for each invoice with minimum columns for the sake of simplicity.

DROP TABLE IF EXISTS invoices;
CREATE TABLE invoices(
    id int identity(1,1) primary key,
    raised_date date not null,
    raised_by int not null,
    customer_id int not null,
    payment_date date not null,
    total_amount numeric(12,2) not null default(0),
    is_paid bit not null default(0)
);

INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 4, 5, '2021-03-03', 71450.00, 1);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 1, 11, '2021-03-03', 543020.00, 1);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 2, 2, '2021-03-02', 514290.00, 0);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 3, 10, '2021-03-02', 1233400.00, 1);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 1, 2, '2021-03-03', 1951707.00, 1);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 3, 7, '2021-03-02', 39549.00, 1);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 3, 10, '2021-03-03', 25500.00, 0);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 4, 3, '2021-03-03', 22950.00, 1);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 4, 9, '2021-03-03', 818274.00, 1);
INSERT INTO invoices (raised_date, raised_by, customer_id, payment_date, total_amount, is_paid) VALUES ('2021-03-02', 2, 1, '2021-03-03', 454338.00, 1);

DROP TABLE IF EXISTS invoice_lines;
CREATE TABLE invoice_lines(
    id int identity(1,1) primary key,
    invoice_id int,
    product_id int,
    unit_count int not null default(0),
    unit_price numeric(8,2) not null 
);

INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (2, 1, 49, 7145);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (1, 1, 10, 7145);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (9, 1, 33, 7145);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (4, 2, 9, 9549);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (9, 2, 7, 9549);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (5, 3, 9, 39549);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (4, 2, 10, 8227);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (5, 4, 54, 4883);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (9, 5, 41, 9549);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (3, 6, 70, 7347);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (8, 7, 18, 1275);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (2, 1, 27, 7145);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (4, 2, 60, 9549);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (9, 5, 13, 9549);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (5, 4, 84, 4883);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (4, 6, 67, 7347);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (5, 8, 88, 9549);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (5, 7, 64, 1275);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (7, 7, 20, 1275);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (10, 9, 18, 25241);
INSERT INTO invoice_lines (invoice_id, product_id, unit_count, unit_price) VALUES (6, 3, 1, 39549);

DROP TABLE IF EXISTS products;
CREATE TABLE products(
    id int primary key,
    item_name varchar(50) not null,
    item_description varchar(100),
    unit_price numeric(8,2) not null default(0)
);

INSERT INTO products(id, item_name, unit_price) VALUES (1, 'Compactor', 7145);
INSERT INTO products(id, item_name, unit_price) VALUES (2, 'Crawler', 9549);
INSERT INTO products(id, item_name, unit_price) VALUES (3, 'Excavator', 39549);
INSERT INTO products(id, item_name, unit_price) VALUES (4, 'Scraper', 4883);
INSERT INTO products(id, item_name, unit_price) VALUES (5, 'Grader', 9549);
INSERT INTO products(id, item_name, unit_price) VALUES (6, 'Trencher', 7347);
INSERT INTO products(id, item_name, unit_price) VALUES (7, 'Backhoe', 1275);
INSERT INTO products(id, item_name, unit_price) VALUES (8, 'Dragline', 9549);
INSERT INTO products(id, item_name, unit_price) VALUES (9, 'Dump Truck', 25241);

DROP TABLE IF EXISTS product_approvals
CREATE TABLE product_approvals(
    id int identity(1,1) primary key,
    product_id int,
    item_name varchar(50),
    item_description varchar(100),
    unit_price numeric(8,2) not null default(0),
    is_approved bit not null default(0)
)

With our table and data set up as we wanted, let's do those few examples.

DML Trigger Examples

FOR Insert Trigger

FOR Insert Trigger will be useful in cases where we want to set up some simple validation or check the existence of data before performing any operations. This Trigger can be used to run some t-SQL statements before the insert operation in the table. In our case of invoices and invoice_lines table, say we want to implement a rule that once the invoice is settled i.e. paid then new items can't be added to that invoice. The below snippet will help to create a trigger for the same.

CREATE TRIGGER TR_F_I_invoice_lines_validate_invoice_id
ON dbo.invoice_lines
FOR INSERT
AS
IF NOT EXISTS (SELECT * FROM invoices t1 JOIN inserted i ON t1.id = i.invoice_id WHERE t1.is_paid = 0)
BEGIN
    RAISERROR('Invoice doesnt exists or is already paid. Hence cant be modified', 16, 1);
    ROLLBACK TRANSACTION;
END
GO

INSERT INTO invoice_lines(invoice_id, product_id, unit_count, unit_price) VALUES (2, 1, 1, 7145.00); -- return error

INSERT INTO invoice_lines(invoice_id, product_id, unit_count, unit_price) VALUES (3, 1, 1, 7145.00); -- perfrom the insert operation on invoice_lines table

When we try to insert an invoice line item for invoice_id 2 it returns and rolls back with the error we have mentioned, but in the case of invoice_id = 3, it passes this validation since invoice_id 3 is not paid yet and it also exists in the invoices table.

AFTER Insert Trigger

AFTER Insert Trigger will be useful where we want to execute some T-SQL code after each insertion we have done in that table. Suppose, in our example case, we want to calculate the total amount each time an invoice item is added to the invoice_lines table, one way to do this is to update the invoice total_amount every time a new item is added to the invoice_lines table for that invoice. This is can be done with the 'AFTER Insert trigger' as with the insert operation done in the invoice_line table, we will update the total_amount value in the invoices tables.

CREATE OR ALTER TRIGGER TR_A_I_invoice_lines_update_invoice_total_amount
ON dbo.invoice_lines
AFTER INSERT
AS
BEGIN
    UPDATE t1
        SET t1.total_amount = t1.total_amount + (t2.unit_price * t2.unit_count)
    FROM invoices t1
    JOIN inserted t2
        ON t1.id = t2.invoice_id;
END
GO

The above code snippet is an AFTER INSERT trigger, wherein we update the total_amount value of the invoice table in case of the new addition of the invoice_items. The below code will show you the example. It updates the total_amount column in the invoices table when that invoice is unpaid, but if it's paid invoice and we try to add new items for it invoice_items table, then it returns with the error from the previous validation trigger we made.

SELECT 
    *
FROM invoices i
JOIN invoice_lines il
    ON i.id = il.invoice_id
WHERE i.id = 3;

INSERT INTO invoice_lines(invoice_id, product_id, unit_count, unit_price) VALUES (3, 4, 1, 4883.00);
INSERT INTO invoice_lines(invoice_id, product_id, unit_count, unit_price) VALUES (3, 2, 1, 9549.00);

SELECT 
    *
FROM invoices i
JOIN invoice_lines il
    ON i.id = il.invoice_id
WHERE i.id = 3;

INSTEAD OF Insert Trigger

Now, we will see a working example for the usage of INSTEAD OF Insert Trigger. It can be helpful in cases where we want to replace the insert operation with some other T-SQL code and execute that code. It mostly is used with views, which is acting as an interface on top of one or multiple table objects for the application server. In our case, we will be using it to change the flow of how the products are added to our catalog i.e. products table. Let's say earlier the company used to directly add the products which they wants to sell directly in the products table. But recently the company has decided to implement an approval process for the products. For every new product being added, it will have to be approved by the company to appear in the products section. In such a scenario we can use INSTEAD OF Insert Trigger to replace the product insert operation in the products table to prdoucts_approvals table. Let's see the trigger for the same.

CREATE OR ALTER TRIGGER TR_IN_instead_products_insert_in_product_approvals
ON dbo.products
INSTEAD OF INSERT
AS
BEGIN
    INSERT INTO product_approvals(product_id, item_name, item_description, unit_price) 
    SELECT i.id, i.item_name, i.item_description, i.unit_price FROM Inserted i;
END
GO

The above trigger definition replaces the insert operation in the products table with the product_approvals table. This way the change in the company product approval process can be altered. Let's try to perform some insert operations.

--select * from products;
--select * from product_approvals;

INSERT INTO products(id, item_name, item_description, unit_price) VALUES (10, 'Pavement Blocks', NULL, 500.78);

--select * from products;
--select * from product_approvals;

With the above example, we can see that the insert operation on the products table is replaced by the product_approvals table. Next, we will be seeing examples of the update operations.

FOR Update Trigger

It is similar to the 'FOR Insert Trigger' we had seen before, but for the update operations. We can have a small validation check or some other T-SQL code executed for each update operation. Earlier in the 'FOR INSERT Trigger' part we created a trigger for validating if the invoice is settled or not and if the invoice is settled, then no new items can be added to that invoice. This trigger won't work in the case of the Update operation of the invoice_lines table. We can test it by updating the invoice_lines table for the invoice_id is 2 and invoice_line_id is 12, even if invoice_id 2 is a paid invoice.

SELECT * FROM invoices WHERE id = 2
SELECT * FROM invoice_lines WHERE invoice_id = 2

UPDATE invoice_lines SET unit_count = unit_count + 1 WHERE invoice_id = 2 and id = 12;

SELECT * FROM invoices WHERE id = 2
SELECT * FROM invoice_lines WHERE invoice_id = 2

Let's prevent this update operation on the invoice_lines table when the invoice is paid and settled. '

CREATE TRIGGER TR_F_U_invoice_lines_validate_for_invoice_id
ON dbo.invoice_lines
FOR UPDATE
AS
IF NOT EXISTS(SELECT * FROM invoices t1 JOIN inserted i ON t1.id = i.invoice_id WHERE t1.is_paid = 0)
BEGIN
    RAISERROR('Invoice doesnt exists or is already paid. Hence cant e modified', 16, 1);
    ROLLBACK TRANSACTION;
END
GO

After creating FOR Update Trigger on the invoice_lines table, let's again try to update the invoice_lines table where the id is 12 and invoice_id is 2. It will return with an error we mentioned in trigger and rollback since the invoice which we were updating was already a settled invoice, but it successfully updates in the case of invoice id 3 as it is the unpaid invoice. You can try with the below SQL snippet.

SELECT * FROM invoices WHERE id = 2
SELECT * FROM invoice_lines WHERE invoice_id = 2

UPDATE invoice_lines SET unit_count = unit_count + 1 WHERE invoice_id = 2 and id = 12;

SELECT * FROM invoices WHERE id = 2
SELECT * FROM invoice_lines WHERE invoice_id = 2

Next, we will see is AFTER Update Trigger;

AFTER Update Trigger

AFTER Update trigger fires and execute the trigger content when the update operation is completed. Note that if the trigger fails, then the update operation also fails. Earlier in 'FOR Update Trigger', we made a trigger to block the updates of the invoices which are already settled, and not of unpaid invoices. Since the unpaid invoices represent that the customer is still in process of shortlisting the items he wants to purchase, we should be able to show the updated total amount of the invoice, every time he updates his invoice items.

CREATE OR ALTER TRIGGER TR_A_U_invoice_lines_update_invoice_total_amount
ON dbo.invoice_lines
AFTER UPDATE
AS
BEGIN
    UPDATE t1
        SET t1.total_amount = t1.total_amount + ((t2.unit_count - t3.unit_count) * t2.unit_price)
    FROM invoices t1
    JOIN inserted t2
        ON t1.id = t2.invoice_id
    JOIN deleted t3
        ON t2.id = t3.id
        ;
END
GO

With the trigger above, every time the invoice_lines table is modified or altered for the invoice id which is not been paid yet, the total amount column in the invoices table will be updated to reflect the new amount.

SELECT * FROM invoices WHERE id = 3;
SELECT * FROM invoice_lines WHERE invoice_id = 3;

UPDATE invoice_lines SET unit_count = unit_count + 1 WHERE invoice_id = 3 and id = 10;

SELECT * FROM invoices WHERE id = 3;
SELECT * FROM invoice_lines WHERE invoice_id = 3;

With every update operation in the invoice_lines table, we update the total amount value in the invoices table, having a column named total_amount.

INSTEAD OF Update Trigger

Similar to INSTEAD OF Insert Trigger, this trigger can be used for a view referencing two or more tables. INSTEAD OF Update Trigger can be used when we want to perform something else instead of the update operation on that table. Let's continue with our example. Earlier we implemented a product_approvals table wherein any new products will have to be approved first by the company before making them available for sale. Similarly, now the company wants to add another flow to it. The change in the price of the product should also be approved by the company before begin passed down to new customers. Let's make a Trigger for that case.

CREATE OR ALTER TRIGGER TR_IN_products_price_update_needs_product_approvals
ON dbo.products
INSTEAD OF UPDATE
AS
BEGIN
    INSERT INTO product_approvals(product_id, item_name, item_description, unit_price)
    SELECT id, item_name, item_description, unit_price FROM inserted
END
GO

When someone tries to update the price of the product, the trigger will instead execute its content. It will insert data in the product_approvals table for that product_id with new column values and set the is_approved flag 0. So it will be reflected on the approvals page and once approved then the changes can be reflected in the products table.

UPDATE products SET unit_price = 7545 WHERE id = 1;

The above update query operation will instead execute the trigger content we created replacing it with inserting the modified row data in the product_approvals table. Now we will be exploring triggers for the DML Delete operation.

FOR Delete Trigger

Similar to FOR Insert Trigger and FOR Update Trigger, FOR delete trigger can be used to perform simple validation before the delete operation, or for some other code execution before the delete operation. Here let's say, before deleting the product from the products table, we have to make sure that there are no pending invoices that have those products in its invoice item list. If the product is in the list then, we have to return an error with a reason for the failed delete operation.

CREATE OR ALTER TRIGGER TR_F_D_product_delete_validate_with_invoice_lines
ON dbo.products
FOR DELETE
AS
IF EXISTS(
    SELECT * 
    FROM invoice_lines il 
    JOIN invoices i
        ON i.id = il.invoice_id
    JOIN deleted d 
        ON il.product_id = d.id WHERE i.is_paid = 0)
BEGIN
    RAISERROR('Product is added in unsettled invoice bill. cant delete at this instance. Try later.', 16, 1);
    ROLLBACK TRANSACTION;
END
GO

DELETE FROM products WHERE id = 6; -- return error

DELETE FROM products WHERE id = 1; -- success

This trigger prevents the deletion of the product which is in the invoice and hasn't yet been settled. Similarly, you can also use FOR DELETE Trigger to validate the delete operation of the already paid invoice.

AFTER Delete Trigger

The AFTER Delete Trigger is used to perform operations when the delete operation is executed. Continuing with our example, let's write a trigger for the case where the invoice item of an unpaid invoice is deleted, then the total amount of that invoice should get updated as well.

CREATE OR ALTER TRIGGER TR_A_D_invoice_lines_update_invoice_total_amount
ON dbo.invoice_lines
AFTER DELETE
AS
BEGIN
    -- select * from deleted;
    UPDATE t1
        SET t1.total_amount = t1.total_amount - (t2.unit_price * t2.unit_count)
    FROM invoices t1
    JOIN deleted t2
        ON t1.id = t2.invoice_id;
END
GO

This trigger updates the total amount of the invoice once the invoice item is deleted. With this, we see the usage of the AFTER Delete trigger. Run the below SQL code to see them working.

SELECT * FROM invoices WHERE id = 3;
SELECT * FROM invoice_lines WHERE invoice_id = 3;

DELETE FROM invoice_lines WHERE id = 23;

SELECT * FROM invoices WHERE id = 3;
SELECT * FROM invoice_lines WHERE invoice_id = 3;

INSTEAD OF Delete Trigger

Continuing our example further, we want to prevent the delete operation on invoices that are already settled and instead return an error. This works similarly to the INSTEAD OF Insert Trigger and INSTEAD OF Update Trigger. Below is the Trigger definition, which prevents the delete operation and returns an error.

CREATE OR ALTER TRIGGER TR_IN_paid_invoice_cant_be_deleted
ON dbo.invoices
INSTEAD OF DELETE
AS
IF EXISTS(
    SELECT * 
    FROM invoices i 
    JOIN deleted
        ON i.id = deleted.id
    WHERE i.is_paid = 1
    )
BEGIN
    RAISERROR('You cant delete an paid invoice.', 16, 1);
    ROLLBACK TRANSACTION;
END
ELSE
BEGIN
    DELETE
        i
    FROM invoices i
    JOIN deleted d
        ON i.id = d.id
    WHERE i.is_paid = 0;
END
GO

In the trigger defined above when the delete operation is executed for the invoices table, we just verify if the invoice we are deleting is already a paid invoice or not. If it's a paid invoice then we prevented the delete operation and instead returned with the error, but if the invoice is unpaid then we can delete that invoice. The below statements help us to verify if the trigger created earlier is working as expected or not.

DELETE FROM invoices WHERE id = 1;

DELETE FROM invoices where id = 7;

-- select * from invoices

This completes our little post regarding the Trigger and its example for all the DML events. In the upcoming post, related to Trigger, we will see how Trigger can be used for multiple or combined DML events like using Trigger to Log Table Data Changes, Nested and Recursive Triggers and DDL Triggers with Events to track database alteration and its example trying to log database changes.

drop table if exists room_series
create table room_series(
    id int identity(1,1) primary key,
    day int,
    start_time time(0),
    end_time time(0),
    time_id int,
    room_id int,
    is_booked bit not null default(0)
);

with weeks (day) as (
    select 1
    union all
    select day + 1
    from weeks
    where day < 7
), five_min_series (start_time, end_time, time_id) as (
    select cast ('00:00:00' as time(0)) as start_time, cast ('00:05:00' as time(0)) as end_time, 1 as level
    union all 
    select
        cast(DATEADD(mi, 5, start_time) as time(0)) as start_time,
        cast(DATEADD(mi, 5, end_time) as time(0)) as end_time,
        time_id + 1
    from five_min_series t1
    where time_id < 288 -- their are 288 block of 5 munities in a 24 hrs.
), rooms (room_id) as (
    select 1
    union all
    select room_id + 1
    from rooms where room_id < 2
)
insert into room_series (day, start_time, end_time, time_id, room_id) 
select day, start_time, end_time, time_id, room_id from weeks, five_min_series, rooms
 option(maxrecursion 0); -- populating records in room_series table

 select * from room_series -- 4032 records

We have created a room_series table that holds records for rooms number 1 and 2 for the 7 days of the week with 5 min granularity. This room_series table contains a column for day reference for a day of the week, start_time, end_time, time_id, and room_id reference for the room number, and the is_booked column default is 0, meaning the particular room_series is not booked and is available for booking or renting as in our case. Now to simulate the situation of finding or validating the request for room booking for a particular day and for a particular time. let's consider that someone has already reserved room number 1 for day 2 and for a start time '08:00:00' am till the end time '16:30:00' pm, we need to simulate this with the below update statement.

update room_series set is_booked = 1 where room_id = 1 and day = 2 and start_time >= '08:00:00' and end_time <= '16:30:00';

Room number 1 is booked or reserved by someone for day 2 and for the time between '08:00:00' and '16:30:00'. Now suppose, we received the asset booking request for room number 1 for the same day i.e day number 2, and was requested for the time between '16:00:00' pm till '21:00:00' pm. With the new request received the request, we need to find that whether his request is overlapping with any other already approved requests. We know that the new request is already overlapping with the booked request for the time frame of '16:00:00' till '16:30:00' pm, due to this half-hour overlap we have to show the user that his request is overlapping/ conflicting and hence canceled. There are 4 cases where we can say that the overlap can occur and they are partial overlap either at the beginning or at the end, complete overlap, and complete within limits, let's consider these scenarios with our illustration, and also for simplicity lets consider the booked room timing as EventA and request for room booking as EventB.

1. When EventB start_time is less than EventA start_time and EventB end_time is greater than or equal to EventA start_time. Essentially when EventB starts before EventA but also ends before EventA. It's a case of partial overlap for the beginning part.

2. When EventB start_time is greater than or equal to EventA start_time AND EventB end_time is less than or equal to EventA end_time. meaning EventB starts at the same time or after EventA starts AND EventB ends before or at the same time as EventA. A case of complete overlap.

3. Similar to case 2 when EventB starts after EventA AND ends before EventA, where the EventB is completely contained within the boundaries of EventA.

4. When EventB start_time is less than or equal to EventA end_time AND EventB end_time is greater than EventA end_time. It means EventB starts before or at the same time as the end of EventA AND EventB ends after EventA. It's a case of partial overlap for the ending part.

But for checking if there is any overlap occurrence, we need to keep in mind two cases.

1. EventB end_time is greater than or equal to EventA start_time.

2. EventB start_time is less than or equal to EventA end_time. Let's check it in our case with the below query

declare @eventB_start_time time(0) = '16:00:00' , @eventB_end_time time(0) = '21:00:00';
select * from room_series where day = 2 and room_id = 1 and start_time <= @eventB_end_time and end_time >= @eventB_start_time and is_booked = 1;

The above result set shows the overlapping or conflicting rows between EventA and EventB. In our case, we knew that there is an overlapping time range from '16:00:00' pm till '16:30:00' pm as eventA is already booked for that time range and eventB also includes that time range in its request for booking, for the same day and the same room number. Note we can adjust the comparison operator as per our requirement in the query. To further optimize, if we don't want to return overlapping rows and show them to the user and only want to check if the user request is overlapping or not. We can use the 'Where NOT EXISTS' clause and the inner query returning the overlapping rows. Below is the snippet.

declare @eventB_start_time time(0) = '16:00:00' , @eventB_end_time time(0) = '21:00:00';
select 1 as is_overlapped
where exists 
(
    select * from room_series where day = 2 and room_id = 1 and start_time <= @eventB_end_time and end_time >= @eventB_start_time and is_booked = 1
);

With this, we return the flag in case we get the overlapping rows between the date or time range as per our requirement.

drop table if exists branch
create table branch(
    id int,
    branch_number int ,
    name varchar(50) not null,
    location varchar(100) not null,
);
create  nonclustered index IX_branch_branch_number  
  on dbo.branch (branch_number, name, location); 

drop table if exists sales
create table sales(
    id int primary key,
    sales_date date not null,
    total_amount decimal(10,2) not null,
    customer_id int not null,
    branch_number int -- we permitted null values
);

insert into branch (id, branch_number, name, location) values (1, 101, 'Douglas-Block', 'Dengyue');
insert into branch (id, branch_number, name, location) values (2, 102, 'Veum-Jones', 'Coyah');
insert into branch (id, branch_number, name, location) values (3, NULL, 'Hansen and Sons', 'Halimpu');
insert into branch (id, branch_number, name, location) values (4, 104, 'Braun-Kemmer', 'Saint Louis');
insert into branch (id, branch_number, name, location) values (5, 105, 'Beatty, Bernhard and Schmitt', 'Dalmacio Vélez Sársfield');
insert into branch (id, branch_number, name, location) values (6, 106, 'Kub, Zboncak and Berge', 'Plyussa');

insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (1, '2021-03-06', 9103.52, 7, 101);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (2, '2021-03-03', 6513.04, 6, 101);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (3, '2021-03-04', 5018.44, 8, NULL);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (4, '2021-03-05', 12187.21, 1, NULL);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (5, '2021-03-03', 8951.13, 8, 102);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (6, '2021-03-04', 2109.0, 8, 105);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (7, '2021-03-05', 4745.6, 7, 104);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (8, '2021-03-05', 5326.41, 7, 106);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (9, '2021-03-08', 3672.56, 5, NULL);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (10, '2021-03-04', 4038.51, 7, 103);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (11, '2021-03-04', 11874.09, 5, 104);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (12, '2021-03-07', 8632.73, 6, NULL);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (13, '2021-03-02', 14909.73, 5, 102);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (14, '2021-03-02', 4014.49, 7, 105);
insert into sales (id, sales_date, total_amount, customer_id, branch_number) values (15, '2021-03-04', 6320.8, 2.3, 106);

Now if we wanted to see the branch details for each sale in the sales table, we need to perform an inner join on both the sales and branch tables.

select * from sales;
select * from branch;

select 
    s.id as sales_id,
    s.sales_date,
    s.total_amount,
    s.customer_id,
    b.name as branch_name,
    b.branch_number,
    b.location
from sales s
join branch b
    on s.branch_number = b.branch_number;

The above image showing the result set of the query doesn't have any data regarding order IDs 3, 4, 9, and 12 as all of these ids have null values. There are ways to join on NULL values, one of which is making those columns NOT NULL and adding default values to them. But what if, we don't have access or permission to change the table structure in the database? We had to make it a workout with what we had.

Using ISNULL OR COALESCE function:

If we cant alter the table structure of the database, then we have to convert those NULL to NON-NULL values using a function like ISNULL or COALESCE. ISNULL function takes two parameters, first is the value of the expression we want to check if it's NULL or NON-NULL and if it's NULL, then it returns the second value we provided for the NULL. While COALESCE returns the first NON-NULL value in the list provided.

select ISNULL(NULL, NULL, 0);
select COALESCE(NULL, NULL, NULL, 1);

The first statement using the ISNULL function returns 0 and the second statement returns 1. Now let's again perform the join operation with the help of these functions for the null column values and see if we get the expected output.

select 
    s.id as sales_id,
    s.sales_date,
    s.total_amount,
    s.customer_id,
    b.name as branch_name,
    b.branch_number,
    b.location
from sales s
join branch b
    on ISNULL(s.branch_number, 0) = ISNULL(b.branch_number, 0);

-- using coalesce
select 
    s.id as sales_id,
    s.sales_date,
    s.total_amount,
    s.customer_id,
    b.name as branch_name,
    b.branch_number,
    b.location
from sales s
join branch b
    on COALESCE(s.branch_number, 0) = COALESCE(b.branch_number, 0);

With the above image, we can see that both statements have returned the required result-set as we wanted. But We have used COALESCE or ISNULL function to get our results. This usage of the function on joining column disables the use of indexes in some databases it does disables index seek. And with a heavy increase in the data set of these tables, the computation cost will rise and will have a performance hit due to computing scaler operation. we can see the flow in the execution plan.

Using OR and IS NULL operator

The below query returns the same result set as the above queries and doesn't add extra computation cost and also enables the index to seek for us on the branch table. While creating the branch table before in the first SQL Snippet, we mentioned branch table columns in the non-clustered index we created.

select
    s.id as sales_id,
    s.sales_date,
    s.total_amount,
    s.customer_id,
    b.name as branch_name,
    b.branch_number,
    b.location
from sales s
join branch b
    on (s.branch_number = b.branch_number OR (s.branch_number is null and b.branch_number is null));

The above query doesn't use any function like the previous two statements above and the left side of the operands mentioning column name is as it is, so it can take the advantage of the indexes. In the above query statement, our branch table performs an index seek retrieval operation. The above approach avoids extra computations and is also performance-friendly for large recordsets. The queries which can take the advantage of the indexes play an important role where the dataset is large and performance is very important.

A Page is a basic unit of data storage in SQL Server and the data files are fully comprised of pages numbered consecutively from 1 to n. The read-write operations that we execute or perform are all made on these pages and they are of various types. An IAM page is a type of page that stores information about the extent (manage space) used by the table or index allocation. It tracks approximately 4GB worth of space in a single file. If a file is more than 4GB or has multiple data file entities, then we need multiple IAM pages to track all the space usage. As some entity requires multiple IAM pages to track their space usage, these IAM pages are linked together and are called IAM chains.

Heap Tables:

In a literal sense, it can be defined as an unordered and unstructured collection of objects placed on top of each other like a pile of books, where books are placed on top of each other without any ordering or anything. Similarly, heap tables are table that holds records or rows without any ordering or any structuring. Since it's just a pile of rows then to find any particular row or record in a table, requires us to perform a scan of rows till we find what we were looking for. This scanning of rows for finding a record in a heap table is known as TableScan Operation. Think of looking for a particular book in a pile of unordered and unstructured books placed on top of each other. And again do note that data order is unpredictable and not guaranteed to retain order. We can add non-clustered indexes to the heap table to speed up the data retrieval process for finding a row. Since the ordering of rows (underlying pages) is absent, heap tables are suitable for heavy data insertions or bulk records. It will just allocate a page and write the data in it and put the page with other pages.

Now with the understanding of the page and IAM page given above, it will be simpler to visualize and understand the logical structuring of the heap table. The heap table is comprised of IAM pages and these IAM pages that are used to manage space allocated to the heap also helps SQL Server to traverse through the heap. The Data Pages and rows within them are not in any specific order and these data pages are not linked to each other. The only connection between these data pages is their information recorded on IAM pages. All reads and writes must contact the IAM first to read all pages within a heap.

Due to the absence of innate ordering or structuring of the pages or data in the heap table, it is suitable for the scenario where heavy insertion activity is required. Each Individual row in the heap table is identified by row identifier (RID). For large unordered insertions, the heap is preferred as we don't need any particular ordering. Thus, insertions in the heap table are generally faster compared to the same insertion in the clustered table. And as discussed earlier, if a heap table does not have any unclustered indexes, then an entire table scan is required to find any row or record.

If we need the data in a particular sorted order frequently, then it is better to not use heap and then sort row sets on return with ORDER BY clause. A Heap table is also not advisable where data grouping or querying for data ranges is frequently made. If a large table without any non-clustered indexes is needed to be queried with or without any particular sorting, grouping, or order condition, in that case, it is suitable to not use a heap. For Frequent data updates, the heap table is not suitable.

Clustered Table

It is an ordered and structured collection of objects. The Clustered index provides underlying ordering and structuring of the data set in the table. The columns for the ordering can be defined at the time of table creation as Primary Key or by defining a clustered index on an existing heap table. To use the earlier example of a pile of books, this can be said as a set of ordered books arranged in either ascending or descending order based on the name of the book or author's name or publish date or combinations of it. As this is an ordered set of books, then finding any particular book requires us to perform seek operation and locate the required book we were finding.

In the clustered table the record is stored in a Binary Tree structure (B-Tree), wherein the anchor point or starting point is a root node and expands to intermediate nodes and leaf nodes till it covers the entire table for index. These nodes provide the pointers to the previous node and next node in a data set and the rows or records are only stored at the leaf nodes which are at the lowest level of the clustered index structure and are ordered by the clustered index columns. Due to this structuring and ordering of the data set, any insert, update or delete operation of the underlying ordering of the data is retained.

As mentioned earlier with its innate ordering, finding a particular row in an ordered table is to seek operations at the required multiple levels and locate the required rows. Every child node it traverses to reach the leaf node, reduces and filters out the rows which are not needed to be scanned or sought. This is why seek operation is only available in the clustered index table and not in the heap table, making the read or retrieval speed of data faster compared to the heap table. We can create clustered indexes on a single column or multiple columns together as a composite key. Each table can only have one clustered index as one order sorting criteria can be used.

Since the rows are ordered and sorted as per the criteria mentioned with the required columns during the creation of the clustered index, the clustered table is suitable for scenarios where heavy read activity is necessary. It stores the data rows sorted based on the clustered index key values.

For heavy bulk insertions, the clustered table is needed to maintain and retain the clustered index while doing insert, update or delete operations. This is because if the data rows which are to be inserted are not in sorted order then, sorting the data set will be done first, which adds up the extra cost. But this need sorting data is not needed while inserting the same data rows in the heap table. Since heap tables don't care about the ordering of the rows. It will only find the space within each page and then write on those pages. Let's try a simple example for analyzing the performance of both clustered tables and heap tables.

-- creates room_series table with clustered index on id column
drop table if exists room_series
create table room_series(
    id int identity(1,1) primary key,
    day int,
    start_time time(0),
    end_time time(0),
    time_id int,
    room_id int
);

with weeks (day) as (
    select 1
    union all
    select day + 1
    from weeks
    where day < 7
), five_min_series (start_time, end_time, time_id) as (
    select cast ('00:00:00' as time(0)) as start_time, cast ('00:05:00' as time(0)) as end_time, 1 as level
    union all 
    select
        cast(DATEADD(mi, 5, start_time) as time(0)) as start_time,
        cast(DATEADD(mi, 5, end_time) as time(0)) as end_time,
        time_id + 1
    from five_min_series t1
    where time_id < 288 -- there are 288 blocks of 5 minutes in 24 hrs.
), rooms (room_id) as (
    select 1
    union all
    select room_id + 1
    from rooms where room_id < 100
)
insert into room_series (day, start_time, end_time, time_id, room_id) 
select day, start_time, end_time, time_id, room_id from weeks, five_min_series, rooms
 option(maxrecursion 0); -- populating records in room_series table

drop table if exists clustered_room_series;
create table clustered_room_series(
    id int not null primary key, -- doesnot have identity(1,1) property
    day int,
    start_time time(0),
    end_time time(0),
    time_id int,
    room_id int
);

drop table if exists heap_room_series;
create table heap_room_series(
    id int not null,
    day int,
    start_time time(0),
    end_time time(0),
    time_id int,
    room_id int
);

For the illustration first, we created a room_series table that holds our data to simulate insertions. Then we created two tables above, one is a clustered table named clustered_room_series and the other begins a heap table named heap_room_series. The clustered table has a primary key on the id column, so it will create a clustered index on the clustered_room_series table. Now we will be performing inserts from a table named room_series where we have 201600 rows as data. It has clustered index on the id column. We will simulate insertions to see how both tables respond.

insert into clustered_room_series(id, day, start_time, end_time, time_id, room_id) select id, day, start_time, end_time, time_id, room_id from room_series;

insert into heap_room_series(id, day, start_time, end_time, time_id, room_id) select id, day, start_time, end_time, time_id, room_id from room_series;

-- Table 'clustered_room_series'. Scan count 0, logical reads 5998, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
-- Table 'room_series'. Scan count 1, logical reads 1132, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

-- (201600 rows affected)

-- (1 row affected)
-- Table 'heap_room_series'. Scan count 0, logical reads 202726, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
-- Table 'room_series'. Scan count 1, logical reads 1132, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

-- (201600 rows affected)

-- (1 row affected)

-- Completion time: 2022-03-24T05:09:40.6912041+05:30

As we can see with the above image of the query plan for both insertions, it can be understood that the cost is the same. Since the room_series table, our source table is already clustered table, it doesn't need to spend any extra to sort the rows before insertion in the clustered_room_series table. There is a difference in logical reads for both insertions. Insertion in clustered table required 5998 logical reads and heap table required 202726 reads.

With a low logical reads count, we can say that clustered table works better than a heap table when the source data set is also properly indexed. Various combination of indexes gives various output. For the table with the clustered index, a new data row must be inserted into the correct location as per the column definition in an index. And in the heap table, the new data row itself can be inserted into any page on which space is available, or on a new page at the end of the chain.

Now let's simulate the above script again but this time the source table i.e. room_series will be a heap table. And we will be performing the insertions in both clustered and heap tables.

insert into clustered_room_series(id, day, start_time, end_time, time_id, room_id) select id, day, start_time, end_time, time_id, room_id from room_series;

insert into heap_room_series(id, day, start_time, end_time, time_id, room_id) select id, day, start_time, end_time, time_id, room_id from room_series;
--Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
--Table 'room_series'. Scan count 9, logical reads 1127, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
--Table 'clustered_room_series'. Scan count 0, logical reads 5998, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

--(201600 rows affected)

--(1 row affected)
--Table 'heap_room_series'. Scan count 0, logical reads 202726, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
--Table 'room_series'. Scan count 1, logical reads 1127, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

--(201600 rows affected)

With the involved statistics and image above. we can see one major difference and it's the change in the process flow. Since this time the source data, i.e. room_series table is not indexed in any way, the insertion operation for the clustered table in clustered_room_series will sort operation after table scan, and then only clustered index insert can take place.

While with the heap table, it does not matter the ordering of the row sets in any case. The operation is fairly straightforward for heap tables, doing table scans and then table inserts. The query cost is also different from the then first scenario where the source table was clustered table. In this second scenario, the query cost for insertion in the clustered table is about 65 % and in the heap table, it is about 35 %. The logical reads have been the same as the first scenario.

With both scenarios' we can say that with different variants we get different statistics to analyze. In the case of logical I/O, the heap table is at disadvantage compared to the clustered table. Space taken by each table index is also a factor to be considered. The statistics change as we make changes and test out with various modifications, for better performance with optimum cost.

We will be exploring these TSQL functions and understanding the differences between them with examples.

CHECKSUM()

The CHECKSUM() function returns the hash index for an expression or over a row. This hash value can be stored in another column for the row in the entire table, and later can be used for comparison in case of update operations or detecting row changes. As with every update operation, the hash value is also updated mostly based on the expression or column inputs.

CHECKSUM ( * | expression [,...n ] ) 
Two lists return the same values if their types, and attribute values are the same and they are specified in the same order. However, the function does not accept values of these types:

1. CURSOR,

2. IMAGE,

3. NTEXT

4. TEXT

5. XML If of same data types ten NULLS are considered as equal for equality operator check. The dash character is ignored for NCHAR and NVARCHAR data types. And trailing white spaces are also ignored.

As mentioned on the Microsoft SQL Server documentation page “we recommend the use of CHECKSUM only if your application can tolerate an occasional missed change. Otherwise, consider using HASHBYTES instead. With a specified MD5 hash algorithm, the probability that HASHBYTES will return the same result, for two different inputs, is much lower compared to CHECKSUM.”

select checksum('test          ');
select checksum('Test')

Returns the same hash value for the above to two checksum statements (assuming that we have a case-insensitive installation of the instance).

BINARY_CHECKSM()

Similar to CHECKSUM(), returns the binary checksum value computed over a row or list of expressions. It is more precise compared to CHECKSUM(), and hence computation overhead is more than CHECKSUM(). Like CHECKSUM(), BINARY_CHECKSUM() value can be stored in another column for the row in the entire table, and later can be used for comparison in case of update operations or detecting row changes.

BINARY_CHECKSUM ( * | expression [ ,...n ] )

As BINARY_CHECKSUM() computation returns the same value as long as the row isn’t modified later. Unlike CHECKSUM(), the collisions will be less in the case of BINARY_CHECKSUM(). Two lists return the same value if both corresponding attributes have the same data type and are equal when compared using the equals (=) operator or more precise byte representation. However, the function does not accept values of these types:

1. CURSOR,

2. IMAGE,

3. NTEXT

4. TEXT

5. XML

The Microsoft SQL Server Documentation page also mentions similar thoughts for BINARY_CHECKSUM "so to detect whether values have changed, we recommend the use of BINARY_CHECKSUM only if your application can tolerate an occasional missed change. Otherwise, consider using HASHBYTES instead. With a specified MD5 hash algorithm, the probability that HASHBYTES will return the same result, for two different inputs, is much lower than BINARY_CHECKSUM."

select BINARY_CHECKSUM('test');
select BINARY_CHECKSUM('Test');

Both above statements return a different value. Do note that BINARY_CHECKSUM() is case-sensitive and it does not ignore dash characters like CHECKSUM().

HASHBYTES()

The HASHBYTES() function is primarily used for hashing values for use within cryptographic situations. It can be also used for comparison or detecting changes in rows. It returns the hash values based on the specified algorithm as input.

HASHBYTES ( '', { @input | 'input' } )

::= MD2 | MD4 | MD5 | SHA | SHA1 | SHA2_256 | SHA2_512

As HASHBYTES is a more accurate hashing function unlike BINARY_CHECKSUM, it also has a high overhead cost for computations as compared to CHECKSUM() and BINARY_CHECKSUM(). As of SQL Server 2016, all the algorithms are deprecated, except SHA2_256 and SHA2_512. It works on all data types and treats NULL the same as CHECKSUM() and BINARY_CHECKSUM().

select HASHBYTES('SHA2_256', 'test');
select HASHBYTES('SHA2_256', 'Test');

It is also case-sensitive like BINARY_CHECKSUM(). Due to overhead costs, it is intensive, but also suitable for change detection. Though it was limited to 8000 bytes in earlier versions. Fortunately, in SQL Server 2016 and later, that limitation was removed. However, it is correct that the more powerful the hashing algorithm is the more remote are changes of the collisions for different inputs, but this also increases the computational and overhead costs of the resources.

Summary

1. We have to make choices as per the requirement and restrictions based on data types.

2. If want cross-platform implementations, then better to use HASH_BYTES().

3. If dash character or case sensitivity does matter then go for BINARY_CHECKSUM() as it also has lower collisions than CHECKSUM().

4. HASH_BYTES algorithm has its computational overhead, so use it only if necessary.

As mentioned in the post above, the post for the 'rowversion' data type to detect and find changes in rows can be found HERE

with cte_expression_1 as (
    cte_query_1
),
cte_expression_2 as (
    cte_query_2 -- (in the FROM clause refers cte_expression_1)
)
select from cte_epression_2 -- outer query

As mentioned in the syntax above, the second CTE definition cte_expression_2 uses the first CTE definition cte_expression_1. That's why there is a comment with cte_query in the second CTE definition cte_query_2. This means that, once a CTE is defined we can freely use it in subsequent CTEs. This is helpful in use cases where it's impossible to use one aggregate function inside another like finding the average maximum, average minimum, an average of an average, group comparisons for reports, etc.

Let's create a table and seed it with data for illustrations. We are having a scenario wherein the cab service company wants to know the average maximum distance their driver rides in each city they provide the service.

drop table if exists city
create table city(
    id int identity(1,1) primary key,
    name varchar(30) not null,
    country_code varchar(5)
);

insert into city (name, country_code) values ('Fanhu', 'USA');
insert into city (name, country_code) values ('Goodlands', 'USA');
insert into city (name, country_code) values ('Furong', 'USA');
insert into city (name, country_code) values ('Guxian', 'JPN');
insert into city (name, country_code) values ('Ar Rahad', 'OMN');
insert into city (name, country_code) values ('Uitenhage', 'USA');
insert into city (name, country_code) values ('Yanglinshi', 'JPN');
insert into city (name, country_code) values ('Mengjia', 'JPN');

drop table if exists driver
create table driver(
    id int identity(1,1) primary key,
    first_name varchar(20),
    last_name varchar(20),
    vehicle_id int,
    city_id int
);

insert into driver (first_name, last_name, vehicle_id, city_id) values ('Cicely', 'Heaker', 1, 5);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Brandy', 'Rizzone', 2, 2);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Hewitt', 'Foden', 3, 2);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Freddie', 'Rohfsen', 4, 8);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Johann', 'McRill', 5, 1);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Arabella', 'MacKeague', 6, 4);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Jonie', 'Greenan', 7, 8);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Alysa', 'Hanselman', 8, 2);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Vale', 'Heathwood', 9, 1);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Kennedy', 'Headon', 10, 7);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Ugo', 'Rapper', 11, 1);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Laverna', 'Harkin', 12, 3);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Washington', 'Kingdon', 13, 8);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Rogers', 'Ferronet', 14, 7);
insert into driver (first_name, last_name, vehicle_id, city_id) values ('Raleigh', 'Norgan', 15, 5);

drop table if exists daily_ride
create table daily_ride(
    date date,
    driver_id int,
    amount_earned decimal(6,2),
    ride_distance decimal(5,2)
);

insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-11', 8, 198.41, 29.67);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-09', 10, 65.82, 23.39);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-03', 8, 187.77, 5.77);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-28', 13, 273.41, 41.34);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-02', 9, 102.92, 94.51);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-02', 1, 74.85, 38.8);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-06', 4, 312.21, 70.7);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-19', 1, 148.68, 59.91);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-21', 14, 167.77, 35.76);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-24', 9, 223.94, 7.49);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-16', 9, 5.88, 6.5);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-19', 13, 38.31, 25.31);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-24', 4, 33.94, 71.32);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-14', 13, 71.26, 94.04);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-23', 8, 152.09, 1.5);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-06', 6, 206.48, 8.88);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-20', 7, 261.01, 97.81);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-03', 1, 119.72, 6.63);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-13', 10, 77.95, 8.86);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-06', 14, 102.9, 36.22);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-17', 7, 263.06, 63.85);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-05', 6, 125.6, 83.41);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-18', 1, 80.19, 81.05);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-18', 13, 349.35, 60.93);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-18', 3, 230.82, 35.77);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-31', 11, 156.39, 4.28);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-03', 10, 376.38, 72.98);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-05', 7, 114.81, 14.06);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-23', 1, 11.3, 75.65);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-21', 5, 398.57, 11.91);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-28', 9, 178.44, 66.02);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-22', 13, 266.44, 12.16);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-27', 13, 382.47, 37.16);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-28', 13, 327.74, 79.3);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-14', 3, 74.7, 26.2);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-03', 9, 157.97, 36.15);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-22', 13, 212.57, 23.22);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-13', 1, 301.57, 36.76);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-30', 5, 31.8, 73.93);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-12', 3, 396.49, 30.46);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-21', 2, 222.17, 57.34);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-12', 3, 363.4, 41.21);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-12', 9, 227.96, 11.48);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-21', 5, 77.59, 71.0);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-14', 8, 275.88, 97.34);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-16', 5, 365.88, 96.7);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-14', 4, 271.33, 59.66);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-18', 3, 383.73, 79.69);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-08', 6, 245.05, 79.13);
insert into daily_ride (date, driver_id, amount_earned, ride_distance) values ('2021-03-14', 6, 328.01, 91.68);

With all the tables and data-set ready, let's dive in further. ' We have table name city, driver, and daily_rides which have respective records about them. Our cab service company wants to know the average total maximum distance traveling for each city they operate in.

with total_distance_ride as (
    select
        sum(r.ride_distance) as total_distance,
        r.driver_id,
        d.city_id
    from daily_ride r
    join driver d
        on r.driver_id = d.id
    join city c
        on d.city_id = c.id
    group by r.driver_id, d.city_id
), max_ride_distance as (
    select
        max(total_distance) as max_distance,
        city_id
    from total_distance_ride
    group by city_id
)

select avg(max_distance) as avg_max_distance, city_id from max_ride_distance group by city_id

As you see the above image which shows the output of the above Nested CTE finds the average total maximum distance of ride occurs in each city. In the first CTE, we find the total distance driven in each city for each date. In the second CTE, we take this total_distance which is the sum of the ride_distance for that date and city and choose the maximum amongst each city_id. Finally, in the outer query, we return the average maximum ride distance across all the cities. After finding the average maximum distance across those cities, No company wants to find the number of days where the average maximum amount earned exceeded 50 USD. For this we first need to find the maximum amount earned on a given date for each city, then with the previous result set, we need to calculate the average maximal amount across all the cities for a given date. finally, we count the number of dates where the average maximum amount exceeded 50 USD.

with max_amount_earned as (
    select
    max(r.amount_earned) as max_earned,
    r.date,
    d.city_id
    from daily_ride r
    join driver d
        on r.driver_id = d.id
    join city c
        on d.city_id = c.id
    group by r.date, d.city_id
), average_maximal_amount as (
    select
        avg(max_earned) as day_average,
        date
    from max_amount_earned
    group by date
)

select count(date) as number_of_days from average_maximal_amount where day_average > 50;

-- output 
number_of_days
--------------
22

(1 row affected)


Completion time: 2022-03-19T09:00:10.2015488+05:30

The above snippet returns the count of days where the average maximum amount earned was more than 50 USD. Now one last example wherein we will be using nested CTE for comparing two groups of the result set. We want to know the average earning of the cities of the USA and other country codes.

with grouping as (
    select 
        id as city_id,
        case when
            country_code = N'USA'
                then N'US Cities'
            else N'Other'
        end as group_name
    from city
), total_amount_earned as (
    select
        r.driver_id,
        group_name,
        sum(r.amount_earned) as total_earned
    from daily_ride r
    join driver d 
        on r.driver_id = d.id
    join grouping g
        on g.city_id = d.city_id
    group by r.driver_id, group_name
)

select
    group_name,
    avg(total_earned) as average_total_earned
from total_amount_earned
group by group_name order by group_name desc;

With the above image, showing the output, we can that the script above returned the average total amount earned based on our city grouping. In the first CTE - grouping, we just only segregated and clubbed cities as per our grouping requirement based on country_code with help of the case statement. We labeled cities having country_code value 'USA' as 'US Cities' and the rest other as 'Other'. In the second CTE - total_amount_earned we computed total_earned which depicts the sum of amount_earned by the driver and combined it with the grouping label we created in the first CTE. Then in the outer query, we computed the group-level average as the average total earned.

Summary

1. Like Subqueries we cant define CTE inside CTE. But we can refer to the previously defined CTE in subsequent CTEs in the FROM clause.

2. Nested CTEs are more useful in cases where we need to compute aggregate functions on several levels and also to compare two or more grouping of rows.

3. We can use a subquery in the outer query part of the CTE script.

Syntax

The most basic syntax of CTE is as follows,

WITH expression_name  ( column_names )  AS (
  CTE_query -- inner query
) --- cte defination

SELECT
  ...
FROM expression_name -- outer query

expression_name: it is an identifier for the CTE. The same expression_name is not allowed in the case of nested or recursive CTE. Any reference to the identifier in the statement uses CTE.

column_names: its list of names for the columns or attributes of CTE is optional. And the number of column_names should match the number of columns in the result set of the CTE_query.

CTE_query: denotes a SELECT statement to populate the common table expression with the result set. Also known as an inner query. In the rest part of the post, we will also refer to it as an inner query for simplicity. If more than one CTE_query statement is mentioned then, it must be joined using set operators namely UNION ALL, UNION, EXCEPT, and INTERSECT This is the most basic version of the CTE definition and its select outer query. We need to provide a name for CTE and define the CTE within the parentheses. After the parentheses, we can select from the CTE as we do with tables.

To illustrate CTE with examples, let's create a table with some data. In the table, we will be having our employees, departments, and department_expenses. As the names suggest, the employees' table holds records of the employee in a company, the departments' table holds the data of all the departments in a company, and department_expenses records the expenditure made by each department.

drop table if exists departments;
create table departments(
    id int identity(1,1) primary key,
    name varchar(40),
    description varchar(100),
    petty_cash_alloted decimal(6,2)
);

drop table if exists employees;
create table employees(
    id int identity(1,1) primary key,
    first_name varchar(30),
    last_name varchar(30),
    department_id int,
    joining_date date
);

drop table if exists department_expenses;
create table department_expenses(
    id int identity(1,1) primary key,
    employee_id int,
    department_id int,
    expense_amount decimal(6,2),
    expense_date date
);

insert into departments(name, petty_cash_alloted) values ('accounting', 324.50), ('human resources', 1000), ('technical support', 780), ('cleaning staff', 890.50), ('legal', 770.60);

insert into employees (first_name, last_name, department_id, joining_date) values ('Kaitlynn', 'Epelett', 5, '2021-03-24');
insert into employees (first_name, last_name, department_id, joining_date) values ('Bernhard', 'Cuttell', 5, '2021-03-27');
insert into employees (first_name, last_name, department_id, joining_date) values ('Zitella', 'Griniov', 4, '2021-03-25');
insert into employees (first_name, last_name, department_id, joining_date) values ('Janelle', 'Raxworthy', 3, '2021-03-12');
insert into employees (first_name, last_name, department_id, joining_date) values ('Isabeau', 'Handsheart', 1, '2021-03-06');
insert into employees (first_name, last_name, department_id, joining_date) values ('Rowland', 'Jewess', 3, '2021-03-02');
insert into employees (first_name, last_name, department_id, joining_date) values ('Yevette', 'Featenby', 1, '2021-03-27');
insert into employees (first_name, last_name, department_id, joining_date) values ('Cammy', 'Silverton', 3, '2021-03-03');
insert into employees (first_name, last_name, department_id, joining_date) values ('Derward', 'Joules', 4, '2021-03-30');
insert into employees (first_name, last_name, department_id, joining_date) values ('Neilla', 'Hincham', 2, '2021-03-25');
insert into employees (first_name, last_name, department_id, joining_date) values ('Seumas', 'Cribbins', 4, '2021-03-13');
insert into employees (first_name, last_name, department_id, joining_date) values ('Star', 'Bladon', 4, '2021-03-07');
insert into employees (first_name, last_name, department_id, joining_date) values ('Britta', 'Waddilow', 5, '2021-03-15');
insert into employees (first_name, last_name, department_id, joining_date) values ('Beth', 'Burch', 2, '2021-03-16');
insert into employees (first_name, last_name, department_id, joining_date) values ('Essa', 'Gebhardt', 2, '2021-03-22');

insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 134.51, '2021-04-20');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 71.18, '2021-03-19');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 756.75, '2021-03-04');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 84.88, '2021-03-11');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 115.41, '2021-03-18');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 56.15, '2021-03-04');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 771.54, '2021-04-27');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 65.9, '2021-04-26');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (5, 1, 281.28, '2021-04-20');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 72.35, '2021-03-09');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (13, 5, 62.28, '2021-04-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 137.1, '2021-04-13');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (2, 5, 61.91, '2021-05-01');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (3, 4, 389.08, '2021-04-27');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 46.78, '2021-03-30');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (6, 3, 81.24, '2021-03-29');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (5, 1, 177.91, '2021-04-26');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (6, 3, 114.13, '2021-03-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 146.22, '2021-04-29');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 144.58, '2021-03-11');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (3, 4, 150.95, '2021-04-19');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 122.52, '2021-03-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (9, 4, 86.35, '2021-04-28');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (6, 3, 148.5, '2021-03-29');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 442.45, '2021-03-21');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 93.41, '2021-03-18');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 51.72, '2021-03-23');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 138.73, '2021-04-10');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (14, 2, 46.89, '2021-03-13');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 551.32, '2021-04-01');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 62.44, '2021-04-28');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (6, 3, 95.08, '2021-04-13');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (14, 2, 56.33, '2021-03-28');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (13, 5, 146.43, '2021-04-08');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (1, 5, 43.98, '2021-03-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (2, 5, 124.71, '2021-03-03');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (13, 5, 41.12, '2021-04-14');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 102.71, '2021-04-22');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (6, 3, 94.0, '2021-03-29');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (14, 2, 46.05, '2021-04-08');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (14, 2, 95.15, '2021-04-04');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 81.07, '2021-03-26');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 112.0, '2021-03-23');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 147.27, '2021-04-18');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 47.96, '2021-05-01');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 129.75, '2021-03-05');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 95.6, '2021-03-19');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 98.88, '2021-03-13');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 91.24, '2021-05-01');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 47.64, '2021-03-14');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (3, 4, 773.39, '2021-04-22');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 115.93, '2021-04-25');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (13, 5, 86.04, '2021-03-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 53.06, '2021-03-07');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (2, 5, 141.01, '2021-04-02');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (6, 3, 41.77, '2021-04-05');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (1, 5, 63.59, '2021-04-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 71.02, '2021-04-29');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 49.68, '2021-04-18');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 909.59, '2021-04-06');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (9, 4, 954.25, '2021-04-25');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (13, 5, 121.98, '2021-03-03');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 108.31, '2021-03-11');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 96.56, '2021-03-04');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (15, 3, 140.92, '2021-03-19');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 239.05, '2021-03-20');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (14, 2, 72.51, '2021-03-27');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (9, 4, 591.58, '2021-03-25');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 646.08, '2021-04-13');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 116.68, '2021-04-22');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 109.93, '2021-04-16');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (1, 5, 42.47, '2021-04-07');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (8, 3, 146.0, '2021-03-19');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 94.43, '2021-04-20');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (9, 4, 152.56, '2021-03-31');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (5, 1, 60.98, '2021-03-23');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 78.66, '2021-03-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (15, 1, 152.08, '2021-03-28');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (13, 5, 58.16, '2021-03-24');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (3, 4, 62.87, '2021-03-09');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (4, 3, 79.71, '2021-04-19');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 249.22, '2021-03-10');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (3, 4, 378.52, '2021-04-23');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 142.84, '2021-03-10');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (2, 5, 63.81, '2021-04-18');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (12, 4, 106.42, '2021-03-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (13, 5, 82.64, '2021-04-19');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (3, 4, 120.39, '2021-04-02');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 244.59, '2021-04-07');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (1, 5, 50.45, '2021-03-10');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (1, 5, 44.8, '2021-03-28');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (6, 3, 104.68, '2021-04-05');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (2, 5, 136.04, '2021-03-05');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (3, 4, 113.32, '2021-03-04');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (11, 4, 128.12, '2021-03-31');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (9, 4, 576.22, '2021-04-27');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (1, 5, 127.25, '2021-03-22');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (10, 2, 129.69, '2021-03-12');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (2, 5, 116.48, '2021-03-24');
insert into department_expenses (employee_id, department_id, expense_amount, expense_date) values (7, 1, 144.76, '2021-04-02');

The above script will create a table and populate with data for the examples in this post.

Basic CTE Example

We wanted to return the department name as department_name, total expenses of the departments as total_expenses for each department

with department_expenditures as (
    select
        d.name,
        sum(de.expense_amount) as total_expenses
    from departments d
    join department_expenses de
        on d.id = de.department_id
    group by d.name
)

select * from department_expenditures;

With the definition of department_expenditure CTE as name and CTE query inside the parenthesis, we return the required data set i.e. name of each department and their total expenses. This is the most basic example of CTE implementation.

CTE with columns

As mentioned earlier the alternative syntax is a CTE definition with a column list before the parenthesis of the CTE query. The only change is that an extra pair of parenthesis is added before the parathesis of the CTE inner query part. The columns list mentioned are the columns we want our CTE definition to have and we need to construct our CTE query based on these columns we are returning. The column name part is optional and not mandatory, as we did in the above illustration. But we need to use it when we have to return aggregate or analytic function values, we want to use it in another nested or recursive CTE, or even when we want to refer to the values of these functions in the outer query part. We can always use an alias as did in the previous example snippet and not use a column list in the CTE definition, either way, is fine. We will update the above code snippets, to use a column list with a CTE definition.

with department_expenditures (department_name, total_expenses) as (
    select
        d.name,
        sum(de.expense_amount)
    from departments d
    join department_expenses de
        on d.id = de.department_id
    group by d.name
)

select * from department_expenditures;

Now let's modify the return required a little, we only wanted to return those departments who had made total expenses more than 75 percent of their allotted petty cash amount. So we wanted to return those departments that made total expenses more than 75 percent of their allotted petty cash amount.

with department_expenditures (department_name, total_expenses) as (
    select
        d.name,
        sum(de.expense_amount)
    from departments d
    join department_expenses de
        on d.id = de.department_id
    group by d.name, d.petty_cash_alloted
    having sum(de.expense_amount) > d.petty_cash_alloted * 0.75
)

select * from department_expenditures;

The above query will return those department names and their total expenses whose total expenses have crossed more than 75 percent of the allotted petty cash to them. In the above case, all department names and their amount will be returned as all departments have crossed the limit of expenditure.

Multiple CTEs

These CTE are separated by commas and commas should not be given at the end of the CTE. The WITH keyword will be mentioned only once during the start of the CTE definition. This comma-separated multiple CTE makes sense when we have to refer to the return of one CTE in another CTE and so on. For multiple CTE we need to do set-based operations like UNION, UNION ALL, etc. Let's see an example. We wanted to show the list of employees who had made expenses of more than 1.5 times the allotted petty cash amount of both the accounting and cleaning departments.

with accounting_department_expenditure (first_name, last_name, department_name, total_expenses) as (
    select
        e.first_name,
        e.last_name,
        d.name,
        sum(de.expense_amount)
    from employees e
    join departments d
        on e.department_id = d.id
    join department_expenses de
        on de.employee_id = e.id
        and de.department_id = d.id
    where d.name = 'accounting'
    group by e.first_name, e.last_name, d.name, d.petty_cash_alloted
    having sum(de.expense_amount) > 1.5 * d.petty_cash_alloted
), cleaning_department_expenditure (first_name, last_name, department_name, total_expenses) as (
    select
        e.first_name,
        e.last_name,
        d.name,
        sum(de.expense_amount)
    from employees e
    join departments d
        on e.department_id = d.id
    join department_expenses de
        on de.employee_id = e.id
        and de.department_id = d.id
    where d.name = 'cleaning staff'
    group by e.first_name, e.last_name, d.name, d.petty_cash_alloted
    having sum(de.expense_amount) > 1.5 * d.petty_cash_alloted
)

select  
    first_name, 
    last_name, 
    department_name, 
    total_expenses
from accounting_department_expenditure
union all
select
    first_name, 
    last_name, 
    department_name, 
    total_expenses
from cleaning_department_expenditure;

After defining both CTEs we will show the result of those, together with the SET operator in the outer query, here we are using UNION ALL. This will output by combining the output of the above two CTE definitions as in the image above. Do remember that it is not possible to create correlated subqueries with CTEs.

Summary

1. CTEs are named temporary sets of rows similar to subqueries.

2. CTEs are placed at the start of the query. They are introduced with WITH.

3. You can skip column definitions if you provide aliases for columns containing aggregates and other function results.

4. Not possible to create correlated subqueries with CTEs.