How to Fix SQL Database Error 8942

Summary:

In this article, we will solve the SQL Database error 8942 in different ways and, I will tell you the best way for solving this error. The error is:

“Server: Msg 8942, Level 16, State 1 Table error: Object ID O_ID, index ID I_ID, page P_ID, Test (TEST) failed. Slot S_ID, offset 0xADDRESS overlaps with the prior row.”.

Continue reading “How to Fix SQL Database Error 8942”

Column encryption, is it table or data specific?

I ran into an interesting problem lately. I have a table that has encrypted data with a key on a column. The data however needs to be a piece of an ETL process and then it has to become unencrypted.

This led me to ask the question: Can I create a table, insert a record, encrypt the record, move the record to various tables, and finally decrypt the record in a new table successfully?

Microsoft calls this column encryption, since this is a property of a table, I’d partially expect for it to be specific to the table that holds the data. The answer is in the MSDN articles if you read it thoroughly however. “Encrypt a Column of Data”. It’s not encrypt the column in the table, nor is it encrypt the column, it’s a column of data that becomes encrypted with a key and cert.

Let’s create a test scenario here and put the idea to the test. I couldn’t find a definitive black and white answer immediately online so I figured we should figure it out ourselves!

-- Create test table source
CREATE TABLE TestETLSource (
	id INT PRIMARY KEY identity(1, 1)
	,FakeData VARCHAR(255)
	,FakeDataEncrypted VARBINARY(128)
	,StartTime DATETIME2(7)
	,EndTime DATETIME2(7)
	)

-- Insert a record
INSERT INTO TestETLSource (
	FakeData
	,StartTime
	,EndTime
	)
VALUES (
	'Testtesttestyo'
	,'2017-01-02 02:02:02.222'
	,'2017-01-04 02:02:02.222'
	)

Results in table

Before we begin messing with the keys and certificates, let’s see what our current setup is.

--Reference queries to find any symmetric keys, asymmetric keys, and Certificates.
SELECT NAME
	,key_length
	,algorithm_desc
	,create_date
	,modify_date
FROM sys.symmetric_keys;

Symmetric Keys

SELECT NAME
,algorithm_desc
FROM sys.asymmetric_keys;

asymmetric keys

SELECT NAME
,subject
,start_date
,expiry_date
FROM sys.certificates;

certificates

My results returned typical MS certificates, nothing I’ve made before. So I’ll go ahead and create a master key and cert.

If you already have a master key or certificate, you can skip this immediate section.

-- Create master key because none exists
CREATE master KEY ENCRYPTION BY password = 'some strong password'

-- Create certificate
CREATE CERTIFICATE FakeCert ENCRYPTION BY PASSWORD = 'some strong password'
	WITH SUBJECT = 'Just some fake data';
GO

Now that we have a master key and certificate, we can create a symmetric key to use that certificate.

-- Create key
CREATE SYMMETRIC KEY FakeKey
WITH ALGORITHM = AES_256 ENCRYPTION BY CERTIFICATE FakeCert;
GO

New cert

Let’s test the key out!

-- Open the symmetric key with which to encrypt the data.  
OPEN SYMMETRIC KEY FakeKey DECRYPTION BY CERTIFICATE FakeCert
WITH password = 'some strong password';

Now that it is working, we can go ahead and encrypt the value in the column FakeData using the symmetric key FakeKey. We’ll save the data in the column FakeDataEncrypted.

UPDATE TestETLSource
SET FakeDataEncrypted = EncryptByKey(Key_GUID('FakeKey'), FakeData);
GO

Modified table

Make sure to close your key in your session when you are done!

CLOSE SYMMETRIC KEY FakeKey;
GO

Now let’s verify the results. First, open the symmetric key with which to decrypt the data:

OPEN SYMMETRIC KEY FakeKey DECRYPTION BY CERTIFICATE FakeCert
WITH password = 'some strong password';

Now let’s select the FakeData and FakeDataEncrypted column. If the decryption worked, the data will match the decrypted value.

SELECT FakeData
,CONVERT(NVARCHAR, DecryptByKey(FakeDataEncrypted)) AS 'Decrypted Data'
FROM TestETLSource;

Incorrect results

Notice how it doesn’t match? This is due to the nvarchar data type.

SELECT FakeData
,CONVERT(VARCHAR, DecryptByKey(FakeDataEncrypted)) AS 'Decrypted Data'
FROM TestETLSource;

Correct results

Much better! Don’t forget to close the key when you’re done again.

-- Close the key now that we are not using it.
CLOSE SYMMETRIC KEY FakeKey;
GO

Now that we are done testing the key and encrypting the data, we can begin with our real test. Let’s create our stage and destination tables.

-- Create test stage table
CREATE TABLE TestETLStage (
Stageid INT PRIMARY KEY identity(1, 1)
,id INT
,FakeData VARCHAR(255)
,FakeDataEncrypted VARBINARY(128)
,StartTime DATETIME2(7)
,EndTime DATETIME2(7)
);

-- Create test destination table
CREATE TABLE TestETLDest (
Destid INT PRIMARY KEY identity(1, 1)
,Stageid INT
,id INT
,FakeData VARCHAR(255)
,FakeDataEncrypted VARBINARY(128)
,StartTime DATETIME2(7)
,EndTime DATETIME2(7)
);

Now we’ll take the record and move it into our staging table to perform a transformation on the data.

-- Insert records into ETL table
INSERT INTO TestETLStage (
id
,FakeData
,FakeDataEncrypted
,StartTime
,endtime
)
SELECT id
,fakedata
,FakeDataEncrypted
,starttime
,endtime
FROM dbo.TestETLSource;

Stage table

Now we’ll perform an update on our stage data.

-- Perform ETL
UPDATE TestETLStage
SET EndTime = (
SELECT getdate()
);

Updated stage table

Now we’ll take that data out of the stage table and move it over to our destination table.

-- Insert into Dest table
INSERT INTO TestETLDest (
Stageid
,id
,FakeData
,FakeDataEncrypted
,StartTime
,endtime
)
SELECT Stageid
,id
,fakedata
,FakeDataEncrypted
,starttime
,endtime
FROM dbo.TestETLStage;

Dest table results

Here’s where the rubber meets the road. Are our changes from the ETL table valid? Can we still unencrypt the correct values from the column now that we’ve changed tables and updated values in the rows?

OPEN SYMMETRIC KEY FakeKey DECRYPTION BY CERTIFICATE FakeCert
WITH password = 'some strong password';

SELECT FakeData
,CONVERT(VARCHAR, DecryptByKey(FakeDataEncrypted)) AS 'Decrypted Data'
,FakeDataEncrypted
,Destid
,Stageid
,id
,starttime
,endtime
FROM TestETLDest;

Successful results

Great success!

-- Close the key now that we are not using it.
CLOSE SYMMETRIC KEY FakeKey;
GO

As my testing has shown, you can encrypt a column, move the data around different tables, and still decrypt it later. The encryption is on the column itself, not the table.

References used:

Home VPN’s, are you updating your IP securely?

Normally I blog about SQL Server, but I have found this (and one more coming) answers to questions that I could not find on the internet. So I had to share!

As many folks who start dabbling in home VPN setups realize or had known prior (I’m a database guy, not a network admin!), you need a DDNS provider. Your home IP will probably change, so if you wanted to connect to your home IP, it may work for a little while, but it may change unexpectedly. Having it update every few minutes with a provider so they can give you a static name to use that maps automatically to your new IP is the solution to keep it working consistently.

There are many benefits to running a home VPN and using a DDNS service, but the benefit I was after is blocking all adds on my phone. Not just blocking them from being seen, but blocking them from even starting to be downloaded. I run a PiHole at home that blocks ads at the DNS level and is my DNS server. So putting my phone on my home network at all times lets me save data AND block ads EVERYWHERE. If that wasn’t great enough, you can access home files and will have a secure connection to use if you are on a WiFi connection you may not trust.

After dabbling and following some fantastic tutorials as seen by Lauren Orsini on ReadWrite.com and Sam Hobbs on Samhobbs.co.uk I made a lot of progress and got to where I needed an application (DDClient) to update my IP to my DDNS provider and I ran into a problem. Sure it was working, my IP was being updated. But how could I be sure that I was connecting over SSL? Was I transmitting my username and password over plain text across the internet? The answer is, yes.

Here’s the format for a typical DDClient config file:

daemon=60
syslog=yes
mail=root
mail-failure=root
pid=/var/run/ddclient.pid .
ssl=yes
use=web, web=myip.dnsdynamic.org
server=www.dnsdynamic.org
login=USERNAME
password=PASSWORD
server=www.dnsdynamic.org, \
protocol=dyndns2 \
YOUR DOMAIN GOES HERE

Looks fine enough right? Well, it’s not.

If you run this:

sudo ddclient -verbose -debug -noquiet -query

You will quickly see you are connecting over HTTP, not SSL. Even though we have the value ssl=yes, it SHOULD be forcing HTTPS, but we can clearly see our connections are in plain text floating around the internet.

use=web, web=loopia address is IPADDRESSISHERE
CONNECT: myip.dnsdynamic.org
CONNECTED: using HTTP
SENDING: GET / HTTP/1.0
SENDING: Host: myip.dnsdynamic.org
SENDING: User-Agent: ddclient/3.8.2
SENDING: Connection: close

A lot of times I solve problems when I come back to them with a fresh brain and a few cups of coffee. So after a reminder about my problem, I was able to figure out that if I add a simple eight more characters, we would no longer fly through internet tubes seen, but we would be secured.

I changed the line use=web, web=myip.dnsdynamic.org to use=web, web=https://myip.dnsdynamic.org and here’s the new output from the query:

use=web, web=loopia address is IPADDRESSISHERE
CONNECT: myip.dnsdynamic.org

The verification of cert '/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA/CN=www.dnsdynamic.org'
failed against the host 'myip.dnsdynamic.org' with the default verification scheme.

THIS MIGHT BE A MAN-IN-THE-MIDDLE ATTACK !!!!

To stop this warning you might need to set SSL_verifycn_name to
the name of the host you expect in the certificate.

CONNECTED: using SSL
SENDING: GET / HTTP/1.0
SENDING: Host: myip.dnsdynamic.org
SENDING: User-Agent: ddclient/3.8.2
SENDING: Connection: close

Success! I’m not too concerned about the cert issue, if you look, it’s the same subdomain listed. So the query above shows you how you can verify you are connecting over SSL and the config file adjustment with HTTPS resolves your SSL connection.

Auditing your SQL Servers

Below is what I recommend to my clients to evaluate the security of their infrastructure quarterly, although more frequently is always better. Typically with this set of scripts, you’ll want to either compile the data into workbooks, flat files, or at least historic tables so you can track trends and changes.

We begin with two queries; one brings back the SQL Server Name and Instance so we know what we are evaluating and then we check the version of SQL Server which in itself has its own security repercussions based on service pack and year.

SELECT @@SERVERNAME AS 'SQL Server Name and Instance'
SELECT @@VERSION AS 'SQL Server Version'

The next script brings us a simple select of SQL Logins on the server. It’s a nice view of what logins exist, is our policy correctly enforced, do they have expired passwords, are the accounts locked out, and other helpful tidbits. Some additional reading on the returned records can be found here: https://msdn.microsoft.com/en-us/library/ms345412.aspx

SELECT  name ,
        Is_Policy_Checked ,
        Is_Expiration_Checked ,
        LOGINPROPERTY(name, 'IsMustChange') AS Is_Must_Change ,
        LOGINPROPERTY(name, 'IsLocked') AS [Account Locked] ,
        LOGINPROPERTY(name, 'LockoutTime') AS LockoutTime ,
        LOGINPROPERTY(name, 'PasswordLastSetTime') AS PasswordLastSetTime ,
        LOGINPROPERTY(name, 'IsExpired') AS IsExpired ,
        LOGINPROPERTY(name, 'BadPasswordCount') AS BadPasswordCount ,
        LOGINPROPERTY(name, 'BadPasswordTime') AS BadPasswordTime ,
        LOGINPROPERTY(name, 'HistoryLength') AS HistoryLength ,
        Modify_date
FROM    sys.sql_logins; 

This next query brings us a high level overview of what the logins have access to instance wide. It’s a great check for who is sysadmin or not.

SELECT  Name ,
        DenyLogin ,
        HasAccess ,
        SysAdmin ,
        SecurityAdmin ,
        ServerAdmin ,
        SetupAdmin ,
        ProcessAdmin ,
        DiskAdmin ,
        DBCreator ,
        BulkAdmin ,
        UpdateDate
FROM    syslogins
ORDER BY name;

We then break this down to how they have permissions. Is it a SQL or Windows account? What type of permissions are adjusted and what are our datestamps on these logins? Those questions are answered with this script, it’s a more granular approach to the query we previously looked at.

;WITH    ServerPermsAndRoles
          AS ( SELECT   spr.name AS login_name ,
                        spr.type_desc AS login_type ,
                        spm.permission_name COLLATE SQL_Latin1_General_CP1_CI_AS AS permission_name ,
                      --  CONVERT(NVARCHAR(1), spr.is_disabled) AS is_disabled ,
                        'permission' AS permission_type ,
                        spr.create_date AS create_date ,
                        spr.modify_date AS modify_date
               FROM     sys.server_principals spr
                        INNER JOIN sys.server_permissions spm ON spr.principal_id = spm.grantee_principal_id
               WHERE    spr.type IN ( 's', 'u' )
               UNION ALL
               SELECT   sp.name AS login_name ,
                        sp.type_desc AS login_type ,
                        spr.name AS permission_name ,
                       -- CONVERT(NVARCHAR(1), spr.is_disabled) AS is_disabled ,
                        'role membership' AS permission_type ,
                        spr.create_date AS create_date ,
                        spr.modify_date AS modify_date
               FROM     sys.server_principals sp
                        INNER JOIN sys.server_role_members srm ON sp.principal_id = srm.member_principal_id
                        INNER JOIN sys.server_principals spr ON srm.role_principal_id = spr.principal_id
               WHERE    sp.type IN ( 's', 'u' )
             ),
        MapDisabled
          AS ( SELECT
          DISTINCT      SP.NAME ,
                        SP.IS_DISABLED
               FROM     SYS.server_principals SP
                        FULL JOIN sys.server_role_members RM ON SP.PRINCIPAL_ID = RM.MEMBER_PRINCIPAL_ID
               WHERE    SP.type IN ( 's', 'u' )
             )
    SELECT  ServerPermsAndRoles.Login_Name ,
            ServerPermsAndRoles.Login_Type ,
            ServerPermsAndRoles.Permission_Name ,
            ServerPermsAndRoles.Permission_Type ,
            ServerPermsAndRoles.Create_Date ,
            ServerPermsAndRoles.Modify_Date ,
            MapDisabled.Is_Disabled
    FROM    ServerPermsAndRoles
            LEFT JOIN MapDisabled ON MapDisabled.name = ServerPermsAndRoles.login_name
    ORDER BY MapDisabled.is_disabled DESC ,
            ServerPermsAndRoles.login_name;

Now we see what those logins are mapped to and what permissions and what databases are impacted. You can see how many people have X, Y, or Z access to your databases and what’s mapped where.

DECLARE @DBRolePermissions TABLE(
 DatabaseName varchar(300), 
 Principal_Name sysname, 
 Login_Name sysname NULL, 
 DB_RoleMember varchar(300), 
 Permission_Type sysname);

INSERT INTO @DBRolePermissions
EXEC sp_MSforeachdb '
 SELECT DISTINCT ''?'' AS DatabaseName, users.Name AS UserName, suser_sname(users.sid) AS Login_Name, 
 roles.Name AS Role_Member_Name, roles.type_desc
 FROM [?].sys.database_role_members r 
 LEFT OUTER JOIN [?].sys.database_principals users on r.member_principal_id = users.principal_id
 LEFT OUTER JOIN [?].sys.database_principals roles on r.role_principal_id = roles.principal_id
   --JOIN [?].sys.database_permissions Action on Action.grantee_principal_id = users.principal_id'

INSERT INTO @DBRolePermissions
EXEC sp_msforeachdb '
 SELECT DISTINCT ''?'' AS DatabaseName, users.Name AS UserName, suser_sname(users.sid) AS Login_Name, 
 r.Permission_Name AS DB_RoleMember, r.class_desc
 FROM [?].sys.database_permissions r 
 LEFT OUTER JOIN [?].sys.database_principals users on r.grantee_principal_id = users.principal_id
 WHERE r.class_desc = ''DATABASE'''

SELECT DISTINCT Principal_Name, Login_Name, DatabaseName, DB_RoleMember AS Permission_Name
FROM @DBRolePermissions 
WHERE  Permission_Type <> 'DATABASE'
ORDER BY Principal_Name, DatabaseName, DB_RoleMember;

After evaluating who has access to what, I like to check and verify what databases are encrypted. This is a quick check to see what is and isn’t encrypted at rest and the strength setup.

USE master;
GO
SELECT db.name,
       db.is_encrypted,
       dm.encryption_state,
       dm.percent_complete,
       dm.key_algorithm,
       dm.key_length
FROM sys.databases db
     LEFT OUTER JOIN sys.dm_database_encryption_keys dm ON db.database_id = dm.database_id;

The last check I do is seeing what the linked server and remote server permissions are like to keep tabs on our other endpoints.

SELECT ss.server_id 
          ,ss.name 
          ,'Server ' = Case ss.Server_id 
                            when 0 then 'Current Server' 
                            else 'Remote Server' 
                            end 
          ,ss.product 
          ,ss.provider 
          ,ss.catalog 
          ,'Local Login ' = case sl.uses_self_credential 
                            when 1 then 'Uses Self Credentials' 
                            else ssp.name 
                            end 
           ,'Remote Login Name' = sl.remote_name 
           ,'RPC Out Enabled'    = case ss.is_rpc_out_enabled 
                                   when 1 then 'True' 
                                   else 'False' 
                                   end 
           ,'Data Access Enabled' = case ss.is_data_access_enabled 
                                    when 1 then 'True' 
                                    else 'False' 
                                    end 
           ,ss.modify_date 
      FROM sys.Servers ss 
 LEFT JOIN sys.linked_logins sl 
        ON ss.server_id = sl.server_id 
 LEFT JOIN sys.server_principals ssp 
        ON ssp.principal_id = sl.local_principal_id;

To bring it all together, it’s important to generate this data as a snapshot in time so you can compare it or write automated reporting to compare off this data so you can see how your environment is changing and who can access what. When in charge of a SQL Server, you need to keep track of who can do what and ensure the proper folks have proper permissions so you don’t run into compliance issues and minimize user impact on the databases.

Who changed my data?

Odds are that if you are reading this; someone, somewhere, somehow… has changed your data. Odds are also, you may be out of luck figuring out what changed and who did it. The good news is that the lesson has been learned and you can take action now to prevent this in the future.

SQL Server by default does not automatically track who does what to your tables. It does track the made to your database, but it won’t tell you “who done it?”. You can however use the transaction logs with extreme painstaking patience to reveal what changes were made.

There are features such as Change Data Tracking and Change Data Capture, this can potentially give you some better insight to the changes as they occur. Unfortunately, when it looks at who made the changes, it is only going to post the principal account running CDC/CDT and it will not reveal the culprit.

CDC / CDT

There is real-time tracing with SQL Profiler, but this creates extreme load on the server and it does not reveal the past. You can utilize SQL’s Dynamic Management Views and Extended Events to capture SQL in real time along with wait types, execution plans, and much more in depth information that doesn’t put as much load on the system. Sp_whoisactive does exactly this! Thank you Adam! So I run this every few minutes to capture data and do analysis on what my long running queries are and what my wait types look like, but more importantly on topic, it captures who ran what query and when. I run other scripts that I will eventually load to github and post for everyone, as well as a more in detailed view on how I performance tune my T-SQL.

The problem with real-time analysis, is that you have to catch it in real-time. There’s the chance the SQL ran too fast or sp_whoisactive was not running at the same time. This leads us to the only real solution, triggers.

I won’t dive into HOW to do this, because that is reinventing the wheel, but I will leave this resource that covers creating an audit trail to figure out what changed and by whom and when.