If you have landed on this blog by searching below error, then I must tell you that your SQL Server service is not getting started. Let us learn in this blog how to fix  The Log Scan Number Passed to Log Scan in Database ‘master’ is not Valid.

2016-01-27 23:36:41.95 spid9s      Starting up database 'master'.
2016-01-27 23:36:42.04 spid9s      Error: 9003, Severity: 20, State: 6.
2016-01-27 23:36:42.04 spid9s      The log scan number (469:72:10) passed to log scan in database 'master' is not valid. This error may indicate data corruption or that the log file (.ldf) does not match the data file (.mdf). If this error occurred during replication, re-create the publication. Otherwise, restore from backup if the problem results in a failure during startup.
2016-01-27 23:36:42.04 spid9s      Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
2016-01-27 23:36:42.04 spid9s      SQL Server shutdown has been initiated

When I was working with my client, all they wanted was to get his SQL Server up and running as soon as possible. I always wish that client has a database backup before the problem. In this case, the master database was corrupt.

The interesting situation here:

• To restore master database, we need to start SQL Server.
• To start SQL Server, we need a master database, which is corrupted.

SOLUTON / WORKAROUND

to move forward, we had to rebuild the master database first. Here is the command which we should use.

setup.exe /QUIET /ACTION=REBUILDDATABASE /INSTANCENAME=MSSQLSERVER /SQLSYSADMINACCOUNTS= /SAPWD

Instance name needs to be replaced based on our installation. For a default instance, its MSSQLSERVER else it should be the instance name. We need to keep in mind that rebuild would rebuild other system databases, msdb and model, as well. It would be safe to take MDF and LDF files of those databases. They can be replaced back once rebuild is complete.

Once rebuild is complete, we need to restore the master database using the query below:

• Start SQL Server in single user mode with -m parameter.
• Use below T-SQL to restore master.

RESTORE DATABASE master FROM disk='C:\backup\master.bak' WITH REPLACE

After rebuilding and restore, we could bring the SQL Server resource online successfully.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – The Log Scan Number Passed to Log Scan in Database ‘master’ is not Valid

I always keep playing with my virtual machine which has SQL Server installed. Recently I was trying to apply the patch of SQL Server and I was surprised to see that SQL Server was not getting started after patching. The error was Error 15240 it cannot write into the file.

I looked into the SQL Server ERRORLOG and found below message just before SQL shut down.

<span style=”font-family: ‘courier new’, courier, monospace;”>
2016-12-21 17:52:15.73 spid5s Exporting ##MS_AgentSigningCertificate## to D:\MSSQL\Data\\MS_AgentSigningCertificate.cer
2016-12-21 17:52:15.73 spid5s Could not export certificate, trying again with random name. If this is a mkmastr build then this is an error.
2016-12-21 17:52:15.73 spid5s Exporting ##MS_AgentSigningCertificate## to D:\MSSQL\Data\\MS_AgentSigningCertificate663E9150-3E68-45A0-8AD8-0FE26C6020B3.cer
2016-12-21 17:52:15.73 spid5s Error: 15240, Severity: 16, State: 2.
2016-12-21 17:52:15.73 spid5s Cannot write into file ‘D:\MSSQL\Data\\MS_AgentSigningCertificate663E9151-3E68-45A0-8AD8-0FE26C6020B3.cer’. Verify that you have write permissions, that the file path is valid, and that the file does not already exist.
2016-12-21 17:52:15.73 spid5s Error: 912, Severity: 21, State: 2.
2016-12-21 17:52:15.73 spid5s Script level upgrade for database ‘master’ failed because upgrade step ‘msdb110_upgrade.sql’ encountered error 15240, state 2, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
2016-12-21 17:52:15.74 spid5s Error: 3417, Severity: 21, State: 3.
2016-12-21 17:52:15.74 spid5s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
2016-12-21 17:52:15.74 spid5s SQL Server shutdown has been initiated
2016-12-21 17:52:15.74 spid5s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.
</span>

One thing which is weird in error log was D:\MSSQL\DATA\\MS_AgentSigningCertificate663E9151-3E68-45A0-8AD8-0FE26C6020B3.cer

Do you see something strange? If not, notice two back slashes after DATA. This is the same folder which contains the master database file. I checked the startup parameters in ERRORLOG and found:

<span style=”font-family: ‘courier new’, courier, monospace;”>
-d D:\MSSQL\Data\\master.mdf
-e C:\MSSQL10.MSSQLSERVER\MSSQL\Log\ERRORLOG
-l E:\MSSQL.MSSQL\mastlog.ldf
</span>

From above, we can confirm that there are two backslashes in the startup parameters when referring to master.mdf file.

SOLUTION / WORKAROUND

Once we removed one of the backslash and started the service without any issues. If you want to know how to modify the startup parameters, then refer to my earlier blog. SQL SERVER – Denali – Startup Parameters Easy to Configure

Have you ever had issues with such incorrect startup parameters?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Fix – Error 15240, Severity: 16, State: 2 – Cannot write into file

One of my clients reported that he is trying to apply security patch for SQL Server clustered instance and it was failing. Since this is SQL 2005 instance, there is no official support available from Microsoft at this point. So, they contacted me. I asked to share setup logs and found a very interesting note.

SQL9_Hotfix_KB2494120_sqlrun_sql.msp.log

MSI (s) (4C!64) [23:29:00:069]: Note: 1: 2262 2: Error 3: -2147287038
MSI (s) (4C!64) [23:29:00:069]: Product: Microsoft SQL Server 2005 (64-bit) — Error 29527. The setup has encountered an unexpected error in datastore. The action is SetInstanceProperty. The error is : Source File Name: datastore\clusterinfocollector.cpp
Function Name: ClusterInfoCollector::collectClusterGroups
Source Line Number: 391
———————————————————-
Failed to determine disk information
Source File Name: datastore\clusterinfocollector.cpp
Compiler Timestamp: Tue Sep 21 15:48:22 2010
Function Name: ClusterInfoCollector::collectClusterGroups
Source Line Number: 390
———————————————————-
Enum*Disks() returned : 2147943568

Interesting message is:  “Failed to determine disk information”

I am not an expert at looking at these logs, but we can clearly see that the setup was failing at getting the disk information. After looking at above error, I asked the DBA to check windows failover cluster manager. He found that there was one disk resource in the available storage that was in the failed state. We can conclude that because of this, the setup was failing.

WORKAORUND / SOLUTION

At this point, we had two choices:

• Bring the disk resource online
• If it can’t come online due to hardware change, then remove it.

They were not able to bring disk online so finally they deleted it. Once they have done that, the setup succeeded for database engine and it was happiness everywhere.

Have you ever encountered an issue with the SQL setup was failing because of another resource?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Apply Security Patch – The setup has encountered an unexpected error in datastore

Along with performance consulting, sometimes I also get requests from a few clients to look at the AlwaysOn availability group related problems.

ENVIORNMENT:

Three node AlwaysOn availability group with multi-subnet cluster with two nodes (SQL-A and SQL-B) within one subnet and the third node (SQL-C) in second subnet.  All three instances are SQL Server 2014 build 12.0.4100.  There is a single Availability Group (ALWAYSONAG) which contains database TESTAG which is shown as not synchronizing.

ISSUE

I have asked more about database and found that the database was TDE enabled. They already followed the steps explained in books online to do backup and restore. But as soon as the database is added to availability group it fails. We found below error message in ERRORLOG and can see that database goes to not synchronizing.

2016-12-23 17:46:41.60 spid55 Starting up database ‘TESTAG’.
2016-12-23 17:46:41.65 spid55 The database ‘TESTAG’ is marked RESTORING and is in a state that does not allow recovery to be run.
2016-12-23 17:46:42.09 spid55 ALTER DB TESTAG with AGNAME:ALWAYSONAG
2016-12-23 17:46:42.09 spid55 ALTER DB param option: SET
2016-12-23 17:46:42.15 spid24s Starting up database ‘TESTAG’.
2016-12-23 17:46:42.16 spid24s Error: 15581, Severity: 16, State: 7.
2016-12-23 17:46:42.16 spid24s Please create a master key in the database or open the master key in the session before performing this operation.

As we can see in the error log there are messages that imply TDE encrypted databases cannot be decrypted by system SPID.

WORKAROUND / SOLUTION

We needed to add encryption by service master key to the database master key. This would allow SQL Server system SPIDs (like recovery spid) to automatically open it as required.

OPEN MASTER KEY DECRYPTION BY PASSWORD = 'Password goes here'
ALTER MASTER KEY ADD ENCRYPTION BY SERVICE MASTER KEY

As soon as we ran the command, the issue was resolved. I think this issue can also happen even with transaction log shipping based log restore also.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – AlwaysOn AG (Availability Group) and TDE Error – Please Create a Master Key

The things I get to learn from my daughter is amazing. I think all of us need to constantly explore without any prejudice and set biases. Whenever I get a new toy for some occasion, I see there is at least a half hour of exploration to understand how the toy works and she goes into a whole new world of searching new ways to play. This makes the whole experience worth it. Let us learn about JSON.

This is almost the same experience I go through working with SQL Server. With SQL Server 2016 introducing JSON, there are a few blogs that I can circle back on using this new capability already. But when you start using the same, the learnings can be multifold. Let us understand from the below code to what the potential value would be:

DECLARE @myjson NVARCHAR(MAX) =
'{
	"Name": "Pinal",
	"Surname": "Kumar",
	"Birth": {"DOB":"2000-12-12" , "Town":"Gandhi Nagar", "Country":"India"}
}'
SELECT JSON_VALUE(@myjson, '$.Name')

As you can see, this is relatively simple code and I am trying to retrieve the value available in the “Name” property. This would return “Pinal” as shown below.

Having said that, if you make a simple mistake of even using a different case or even if you try to retrieve a property that is not existing, the output would be NULL.

SELECT JSON_VALUE(@myjson, '$.name')

SELECT JSON_VALUE(@myjson, '$.name123')

In the above example, both the statements will return you NULL. As I try to wrap up this blog, I want to see how you would solve the above problem? How will you know it was an invalid property or a case sensitive problem? Write it over comments and share your experience.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Understanding JSON Use is Case-Sensitive

Often when you write a large piece of SQL code, it eventually becomes a bit disorganized, or when a big team of developers is working on one project, the code differs between individual developers, the code formatting throughout the project gets inconsistent and hard to read. Let us learn about how to Beautify SQL Code with dbForge SQL Complete.

In this article, we will talk about how to keep your code clean and well-formatted no matter how many developers were involved in writing it. The article also discusses the way to introduce a unified SQL code formatting standard for across a team so that the code could be easily reviewed and edited.

How to make SQL code readable

In a code review process, a reviewer can encounter various coding styles used by individual developers. This can complicate the review process and the reviewer will have to scroll up and down the code many times to completely review the code.

In this case, the code needs to be properly formatted and consistent throughout the file. One of the best SQL code formatter tools I know is dbForge SQL Complete – a smart add-in for Microsoft Visual Studio and SSMS. The tool provides an excellent functionality that can make your code look clean and pretty.

Now, let’s see the tool in action.

The following piece of unformatted messy code is taken from a production environment. You can see that the text is badly formatted lacking indentations and spacing; keywords are presented in a different case throughout the code. In other words, it requires thorough reworking.

When you look at this code, you really don’t know where to start. Luckily, dbForge SQL Complete offers SQL Code Formatting feature. Simply use a well-known key combination (Ctrl+K+D) to quickly make the code properly formatted. The result looks fantastic:

How to customize formatting profiles

The above formatting is a standard formatting pattern used by dbForge SQL Complete. However, if you want to have the code formatted in your own particular way that meets your preferences, then you can manually adjust formatting options. Simply go to the SQL Complete menu and click Options. In the left pane, click Formatting and then click Profiles. You will see three built-in profiles: Default, Profile 1, and Profile 2, each having different formatting options to choose from.

If none of them works for you, you can add your own formatting profile and customize it as you desire. To do this, click New Profile, specify the name and the profile to copy settings from, and then click OK. After that, select the new profile and click Edit Profile.

The Edit Formatting Profile wizard provides a huge number of flexible customizations settings. The wizard is divided into two panes. The left pane presents a tree of features that can be customized individually. When you select a feature, the right pane shows a code sample where different properties for this feature can be set.

For example, you want to introduce the following formatting options:

• put each statement on a separate line;
• add a line break after WHERE statement;
• add a small indent to conditions.

To achieve this, complete the following actions:

• Navigate to the Stacked List option, and clear the Place comma before item in stacked list item;
• Navigate to the Subquery Clear the Use same formatting as in SELECT item and select Line break after opening brace;
• Navigate to the WHERE option and select the Line break after WHERE item;
• Navigate to the FROM option and select the Line break after FROM

Once you are all set, click OK to save the settings. Now, format the code once again and assess the output:

How to share code formatting settings

Now the code looks well-formatted and you can easily read it and edit it.

dbForge SQL Complete allows you to distribute code formatting settings among all members of your team. This way, you can standardize the code appearance so that developers working on a single project could easily read and edit any part of the project’s code.

Conclusion

The described features of dbForge SQL Complete are very helpful when you want your code to be properly formatted and readable. It also provides flexible means for the developers to share code formatting profiles across their team.

Besides, dbForge SQL Complete offers Code Completion, Code Snippets, SQL Refactoring, and a bunch of other useful features that boost up SQL code writing and productivity.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Beautify SQL Code with dbForge SQL Complete

When you test an application that implements billing features, meaningful test data is extremely important. This article teaches how to quickly obtain and populate your database with live currency exchange rates in a few simple steps. Let us learn about Generate Live Currency Exchange Rates.

Before diving into the topic in more detail, let’s drop a line about the meaning of the term currency exchange rate and why it is so important in various aspects of our life.

Currency exchange rate is a reference value used for converting money from one currency to another. In other words, the exchange rate is the price for which one national currency can be exchanged for another national currency.

Whether you are a businessman or the owner of a multi-national corporation, currency exchange rates either directly or indirectly affect your business activities for sure. Currency rates can vary quite dramatically even within one day, so having an up-to-minute information on currency values can save you a good deal of money and help you mitigate risks.

Suppose you need to test a corporate database and you need to populate one of the tables with today’s currency exchange rates. This would seem a challenging task if you were to retrieve data from some source and enter them into the table by hand.

Data generation tool

Luckily, there exists a tool that can provide real-life test data in a few seconds – dbForge Data Generator for SQL Server. This useful sql generator tool includes a great number of built-in data generators that will help you generate meaningful test data of diverse types and for various purposes.

Let’s use the AdventureWorks database to show how real currency exchange rates can be generated by dbForge Data Generator for SQL Server.

There is the Sales.CurrencyRate table in the AdventureWorks database that has the following columns:

• Average Rate — an average exchange rate on a specific day.
• EndOfDayRate — a final exchange rate on a specific day.

After you launch dbForge Data Generator for SQL Server and connect to the AdventureWorks database, select the Sales.CurrencyRate table in the left-hand pane and you will see a preview of the data to be generated by default.

Of course, you can adjust generation settings to obtain data that meet your specific needs, but the resulting data in the AverageRate column still do not resemble the real currency rates, and more importantly they are not live and real.

Select the right generator

This problem can be easily solved by applying Python generator available in dbForge Data Generator for SQL Server. The Python generator allows you to create your own scripts to generate any custom data you need.

JSON API is another trick we will use to get real-time currency rates for our AverageRate column. There are plenty of such solutions within the boundlessness of the internet. Let’s use fixer.io in our example.

So, instead of the predefined generator automatically assigned for the AverageRate column, select the Python generator and insert the following Phyton script into the text box:

import clr
clr.AddReference("System")
import urllib, json
from urllib2 import urlopen
from System import DateTime

def main(config):
    dtStr = str(CurrencyRateDate)
    dt = DateTime.Parse(dtStr)
    year = dt.Year
    month = dt.Month
    day = dt.Day
    n1 = str(FromCurrencyCode)
    n2 = str(ToCurrencyCode)

    if not n1 or not n2:
       return "N/A"

    url = "http://api.fixer.io/"+ str(year) +"-"+ str(str(month).zfill(2)) +"-"+ str(str(day).zfill(2))+ "?base="+str(n1)+ "&symbols="+ n1+","+n2

    response = urllib.urlopen(url)
    data = json.read(response.read())

    if not data.has_key("rates"):
       return "N/A"

    return round(data["rates"][n2],2)

Do the same with the EndOfDayRate column. The Python script, in this case, is as follows:

import clr
clr.AddReference("System")
import urllib, json
from urllib2 import urlopen
from System import DateTime

def main(config):
    dtStr = str(CurrencyRateDate)
    dt = DateTime.Parse(dtStr)
    year = dt.Year
    month = dt.Month
    day = dt.Day
    n1 = str(FromCurrencyCode)
    n2 = str(ToCurrencyCode)

    if not n1 or not n2:
       return "N/A"

    url = "http://api.fixer.io/"+ str(year) +"-"+ str(str(month).zfill(2)) +"-"+ str(str(day).zfill(2))+ "?base="+str(n1)+ "&symbols="+ n1+","+n2

    response = urllib.urlopen(url)
    data = json.read(response.read())

    if not data.has_key("rates"):
       return "N/A"

    return data["rates"][n2]

As easy as pie. Notice that the values in the AverageRate column are exactly what we wanted them to be.

Summary

As is evident from the foregoing, dbForge Data Generator for SQL Server opens great capabilities when it comes to generating large amounts of meaningful realistically looking data for various types of databases.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Generate Live Currency Exchange Rates using dbForge Data Generator for SQL Server

SQL Server troubleshooting topics seem to be the most searched for in this blog. In continuation to this, blog for today is also inspired with one such error message. Some time back I have written below blog which talks about the upgrade script mode of SQL Server: SQL SERVER – Login Failed For User – Reason Server is in Script Upgrade Mode

One of the readers reached out to me with below error in ERRORLOG

spid9s Script level upgrade for database ‘master’ failed because upgrade step ‘sqlagent100_msdb_upgrade.sql’ encountered error 2714, state 6, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.

The interesting errors are:

• There is already an object named ‘DatabaseMailUserRole’ in the database.
• CREATE SCHEMA failed due to previous errors

If we look at sqlagent100_msdb_upgrade.sql which is mentioned in ERRORLOG, below is the place of failure.

EXECUTE msdb.dbo.sp_droprole @rolename = N'DatabaseMailUserRole'
EXECUTE msdb.dbo.sp_addrole @rolename = N'DatabaseMailUserRole'

WORKAROUND/SOLUTION

Looking at error message, we have some issues with DatabaseMailUserRole schema.

1. In SSMS, we went to system databases > msdb > Security > Roles > Database Roles but we didn’t see DatabaseMailUserRole under it. So we ran the following query to double check and no result was returned:

SELECT *
FROM msdb.dbo.sysusers
WHERE (name = N'DatabaseMailUserRole')

So, we verified that “DatabaseMailUserRole” didn’t exist in the MSDB.

2. But if we run the following query to check the schema we found the “DatabaseMailUserRole” exists:

select * from sys.schemas

3. From above, we could see the “DatabaseMailUserRole” schema is indeed orphaned now as no user is associated with it.

At this point, we have two options to resolve this issue:

• Drop the “DatabaseMailUserRole” schema
• Create the “DatabaseMailUserRole” user and made it as the owner of “DatabaseMailUserRole” schema

We used second option to fix the orphaned schema by the script below:

USE [msdb]
GO
CREATE ROLE [DatabaseMailUserRole] AUTHORIZATION [dbo]
GO
USE [msdb]
GO
ALTER AUTHORIZATION ON SCHEMA::[DatabaseMailUserRole] TO [DatabaseMailUserRole]
GO

Now, bigger question is how do we do it if SQL is not getting started? Well, have a look at trace flag 902. Here is the way to start SQL with trace flag. In below, SQL2014 is the name of my instance.

NET START MSSQL$SQL2014 /T902

Have you encountered script upgrade failure? How did you fix them?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Script level upgrade for database ‘master’ failed – There is already an object named ‘DatabaseMailUserRole’ in the database

Some organizations use SQL Server Always On Availability Groups to help boost performance of applications where the users are geographically dispersed. They accomplish this by placing the secondary database in the same facility as the remote users. While it is almost always a good idea to place the data near the application, choosing Always On may be using a sledge hammer to do a screwdriver’s job, given the very high cost of Always On from a software, hardware, and implementation perspective. Of course, RTO (Recovery Time Objective) and RPO (Recovery Point Objective) are prime factors in deciding whether you need Always On, but these factors are related to recovery and not performance. Many organizations overstate their RTO/RPO requirements as most companies are not under the tight requirements of an airline reservation system or financial trading system. In fact, transactional or merge replication may be just the tool you need to meet both the application performance and recoverability requirements of your organization. Let us learn about When to Use a Sledgehammer and When to use a Screwdriver.

I have seen some great examples of companies using SQL Server replication to improve both performance and recoverability. These companies implement the following use cases:

• A website performance monitoring company uses replication to bring data from points of presence around the world to a central database used for reporting and analysis.
• An energy company uses replication to move data to and from remote locations which improves performance for end-users and keeps the main office synchronized with its remote locations.
• A residential security company replicates data to/from residential neighborhoods via SQL Server Express with a central SQL Server to ensure great performance in the neighborhoods while also having the data centrally located for recoverability and reporting purposes.

A common issue when using replication over long distances is that it can fall hopelessly behind. I have seen many companies leverage NitroAccelerator from Nitrosphere to mitigate this issue by attaining near gigabit LAN speeds over these high-latency connections. As a result, they outperform the Always On feature at a fraction of the price. NitroAccelerator installs with no configuration or downtime on each of the replication servers. A cool benefit of NitroAccelerator is it also works on the clients, allowing you to accelerate the end-users’ connections to SQL Server. Therefore, with NitroAccelerator, you can choose the right solution for you in terms of cost and complexity. You can first try directly accelerating the end-user connections, and, if this is not sufficient, then try replication to the remote locations. In both cases, NitroAccelerator will help you ensure the processes are as fast as possible without requiring costly hardware, software, or bandwidth upgrades.

There are always trade-offs when making decisions on performance and recoverability. Replication may be a forgotten tool in your toolbox to address these items at a lower cost and with just as much effectiveness. Perhaps it’s time you use a screwdriver rather than sledgehammer.

For more information on Nitrosphere and/or to try NitroAccelerator, please visit http://nitrosphere.com/trial/

First appeared on SQL Server – When to Use a Sledgehammer and When to use a Screwdriver

The whole concept of InMemory OLTP has been around for a while and still there are areas that I try to go back to learn every single time. If you are new to InMemory OLTP, I would highly recommend searching this blog for more content. For a starter, the blow blog is a great start. SQL SERVER – Beginning In-Memory OLTP with Sample Example

In exploring the DMVs which are available in standard tables, I stumbled upon a great addition. I found the db_stats_properties DMV, which was able to give me information about the number of rows that have been modified.

Getting curious, I wanted to know if this worked with InMemory OLTP tables. The amount of changes and modification to a memory-optimized table is now reflected in the Dynamic Management Function “sys.dm_db_stats_properties” which returns a record per stats object on the table. The DMF is now behaving equally for memory-optimized and disk-based table and the column – “modification_counter” reflects the row modification counter that is used for determining whether an auto – update of stats is needed.

Consider the following query to analyze statistics and the modification_counter value:

USE [MyDatabase]
GO
SELECT
    sp.stats_id, name, filter_definition, last_updated, rows, rows_sampled, steps,
unfiltered_rows, modification_counter
FROM sys.stats AS stat
CROSS APPLY sys.dm_db_stats_properties(stat.object_id, stat.stats_id) AS sp
WHERE stat.object_id = object_id('Schemaname.TableName');

One of the things to keep in mind is that the data on the DMF resets on DB restart or failover. A typical output would look like:

It is worth to know that the modification_counter can be more than the number of rows. The logic is, if the same rows are modified multiple times, then this counter can be higher than the actual number of rows.

Do let me know if you find this interesting. Where would you use this capability? Let me know via comments below.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Using dm_db_stats_properties With InMemory OLTP Tables

One of my client was trying to deploy his product on their test servers. They found that the SQL Server installation was failing with below error. Let us learn about error Wait on the Database Engine recovery handle failed.

Wait on the Database Engine recovery handle failed. Check the SQL Server error log for potential causes.

As per my internet search this happens when SQL Setup is not able to start SQL Service during installation. I have looked into SQL Server ERRORLOG and found below the messages.

2016-12-23 16:15:04.84 spid14s Error: 17120, Severity: 16, State: 1.
2016-12-23 16:15:04.84 spid14s SQL Server could not spawn FRunCommunicationsManager thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.

This was interesting because my client was trying to install this on a Domain Controller. I checked setup logs again and found below in Summary.txt file.

SQLSVCACCOUNT: NT Service\MSSQLSERVER

Above parameter shows the account name which was selected during installation.  Since this is a domain controller, I thought that there might be issues due to this.

WORKAROUND / SOLUTION

As the error says, first we need to uninstall the feature and then during setup, when we come to Server Configuration Screen, we need to change the account name of the Database Engine Service to

1. NT AUTHORITY\NETWORK SERVICE.
2. NT AUTHORITY\LOCAL SERVICE.

Both worked for my client and they could install SQL Express.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Installation fails with error code 0x851A001A – Wait on the Database Engine recovery handle failed

Of late, I have been contacted by few clients who have been facing issues with SQL startup failure after applying patch to SQL Server instance. This blog is an outcome of one such assistance provided.  Here is a blog post where I discuss script level upgrade for database master fails.

Once the patch was applied, SQL service was not able to start. So, I asked for SQL Server ERRORLOG SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

Here is the relevant information.

spid5s Cannot bulk load. The file “C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Install\SqlTraceCollect.dtsx” does not exist.

Whenever there is an upgrade failure, we always see “Cannot recover the master database. SQL Server is unable to run.”  This is a side effect of upgrade failure.

WORKAROUND/SOLUTION

From the first error message, it is clear that we are missing files SqlTraceCollect. dtsx from C:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Install So there are two ways to fix it.

1. Get the file from the server which has exactly the same build of SQL Server installed successfully.

This would be a quick fix, but we may have some more files missing which we can never figure out.

2. The right thing to do at this point is to uninstall the patch which we last applied and then “Repair” the instance. Here is the option in setup installation center.

If it is repaired successfully, we should get the files back and then we can apply the patch again.

Have you ever seen any other errors in script upgrade mode?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Script level upgrade for database master failed – Error: 4860, Severity: 16, State: 1 – Cannot bulk load. SqlTraceCollect.dtsx does not exist

Playing around with the new capability of SQL Server 2016 has been a fun journey. Even after a year, the discovery keeps happening every single day that I work with it. When I wrote the post around NULL values getting returned, lesser did I know there have been folks who work on this feature.  Let us learn in this blog post about JSON NULL Value Using STRICT Keyword. Do make sure to read the other blog too: SQL SERVER – Understanding JSON Use is Case-Sensitive

Some of the answers provided got me interested in posting the next in the series. This was a learning for me too. While using JSON, there is a keyword that can-do magic, it is called as STRICT. The same code which was earlier used in the blog has been modified to use this keyword. The same code would look like:

DECLARE @myjson NVARCHAR(MAX) =
'{
	"Name": "Pinal",
	"Surname": "Kumar",
	"Birth": {"DOB":"2000-12-12" , "Town":"Gandhi Nagar", "Country":"India"}
}'
SELECT JSON_VALUE(@myjson, 'strict $.name')

Msg 13608, Level 16, State 5, Line 1
Property cannot be found on the specified JSON path.

As you can we no longer get the NULL value as discussed before. We are getting an explicit error that represents the actual problem.

While I was trying to play around with the NULL value, I figured out when the value returned string is larger than VARCHAR (4000), then there would be an error while using the STRICT keyword.

Msg 13625, Level 16, State 1, Line 1
String value in the specified JSON path would be truncated.

The learning never stops working with SQL Server. Do let me know how you are using JSON datatypes? Let us know via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Understanding JSON NULL Value Using STRICT Keyword

SQL Server 2016 brought several innovations. One of the most anticipated feature additions was the ability to define foreign keys with InMemory tables. I wrote about this feature earlier on blog at: SQL SERVER 2016 – InMemory OLTP support for Foreign Key.

Though this was a great addition, I always felt there were restrictions still to know and understand. The twist to the previous blog was to create a Foreign key on a standard disk based table. I went about using the below script for the same.

CREATE TABLE Products
(
ProductID INT CONSTRAINT pk_products_pid PRIMARY KEY NONCLUSTERED HASH (ProductID) WITH (BUCKET_COUNT = 10000),
ProductName VARCHAR(25)
 ) WITH ( MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);
GO
CREATE TABLE ProductSales
(
SalesID INT CONSTRAINT pk_productSales_sid PRIMARY KEY,
ProductID INT CONSTRAINT fk_productSales_pid FOREIGN KEY REFERENCES Products(ProductID),
SalesPerson VARCHAR(25)
 )
GO

I was pleasantly surprised that an error message popped up. I am showing the same below:

Msg 10778, Level 16, State 0, Line 1
Foreign key relationships between memory optimized tables and non-memory optimized tables are not supported.

This was a great learning that was worth a share via blogs. Though the foreign key addition was lesser known to many, even more lesser known is the fact that it cannot be across disk based tables and in-memory tables. Do let me know if you were aware of this?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error Msg 10778, Level 16 with InMemory OLTP

Have you ever been in a situation where SQL Server shuts down by itself? It sounds strange but I met across a gentleman in flight who reported this issue to me. Since I was curious to know the cause, we exchanged our business cards. Once I came back home, I sent him an email asking more details about the SQL shutdown issue. Let us learn about how to fix error, An Error Occurred While Writing an Audit Trace.

I asked him to share ERRORLOG from the SQL Server instance and I could find below information just before the shutdown.

spid397 Error: 566, Severity: 21, State: 1.
An error occurred while writing an audit trace. SQL Server is shutting down. Check and correct error conditions such as insufficient disk space, and then restart SQL Server. If the problem persists, disable auditing by starting the server at the command prompt with the “-f” switch, and using SP_CONFIGURE.

The error is kind of self-explanatory. SQL was shut down because it was not able to write to trace file, but the real question is why?

One of the possible way this error would come up would be if we were doing C2 type auditing.

But in the above case we can see that it’s set to zero. Then I have asked for more details about various traces which were running and he provided me below script.

exec @rc = sp_trace_create @TraceID output, 6, N’\\IndiaFileServer\T$\MSSQL_Trace\ProdServers\FooBar_Trace’, @maxfilesize, NULL

As per books online, the 6 above is a combination of options 2 & 4.  Option 2 makes file rollover after they reach @maxfilesize.  Option 4 specifies that if SQL ever can’t write to the trace file, it will shut down the SQL Server.  This is by design for a security auditing feature.

To find if shutdown option is enabled or not, we can look at running traces using catalog view called as sys.traces.

Since we know that trace is configured to shutdown SQL server in case of failure to write. He agreed to change option to use 2 instead of 6.

At the end, I was happy that we could find cause and solve a mystery.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error: 566, Severity: 21 – An Error Occurred While Writing an Audit Trace. SQL Server is Shutting Down

I was trying to set up database mirroring in my freshly created Azure Virtual Machines and encountered an error. The error was talking about the connectivity problem, but it was something else. Let us learn about how to fix the database mirroring connection error 4.

Environment

• SRV_P -Principal- Microsoft SQL Server 2014 – 12.0.4422.0 (X64)
• SRV_M-Mirror- Microsoft SQL Server 2014 – 12.0.4422.0 (X64)
• SRV_W-witness- SQL SERVER 2008 R2

SYMPTOMS

As soon as I configure database mirroring and click on “Start Mirroring”, I was getting below error.

Alter failed for Database ‘SQLAUTHORITY’. (Microsoft.SqlServer.Smo)
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
The ALTER DATABASE command could not be sent to the remote server instance ‘TCP://SRV_W.SQLAUTHORITY.net:5022’. The database mirroring configuration was not changed. Verify that the server is connected, and try again. (Microsoft SQL Server, Error: 1456)

I looked around in ERRORLOG files to see if there is anything interesting.

PRINCIPAL SERVER

2016-12-26 01:23:16.710 spid43s Database mirroring connection error 4 ‘An error occurred while receiving data: ’24(The program issued a command but the command length is incorrect.)’.’ for ‘TCP://SRV_W.sqlauthority.net:5022’.
2016-12-26 01:23:36.490 spid75 Error: 1456, Severity: 16, State: 3.
2016-12-26 01:23:36.490 spid75 The ALTER DATABASE command could not be sent to the remote server instance ‘TCP://SRV_W.sqlauthority.net:5022’. The database mirroring configuration was not changed. Verify that the server is connected, and try again.
2016-12-26 01:23:36.750 spid41s Database mirroring is active with database ‘SQLAUTH’ as the principal copy. This is an informational message only. No user action is required.

MIRROR

2016-01-01 22:28:14.370 Logon Database Mirroring login attempt by user ‘SQLAUTHORITY\SRV_P$.’ failed with error: ‘Connection handshake failed. The login ‘SQLAUTHORITY\SRV_P$’ does not have CONNECT permission on the endpoint. State 84.’. [CLIENT: 10.0.40.34]

WORKAROUND/SOLUTION

I realized that this is a typical issue where service account is set to LocalSystem. Due to this mirroring, would use the machine name (having $ symbols at the end) to communicate. There are two workarounds for this.

1. Change the Service account to domain account
   Read more here: SQL SERVER – Best Practices About SQL Server Service Account and Password Management

2. If you don’t want to use domain account, then we need to create machine account as login and provide CONNECT permission to endpoint (as mentioned in error message)

On principal

use [master]
GO
CREATE LOGIN [SQLAUTHORITY\SRV_M$] FROM WINDOWS
GO
GRANT CONNECT ON ENDPOINT::[Mirroring] TO [SQLAUTHORITY\SRV_M$]
GO
use [master]
GO
CREATE LOGIN [SQLAUTHORITY\SRV_W$] FROM WINDOWS
GO
GRANT CONNECT ON ENDPOINT::[Mirroring] TO [SQLAUTHORITY\SRV_W$]

Notice that we are giving permission to machine account of mirror and witness. Same set of command to be run on Mirror and Witness as well with other two machines account.

On Mirror = SQLAUTHORITY\SRV_W$ and SQLAUTHORITY\SRV_P$

On Witness = SQLAUTHORITY\SRV_P$ and SQLAUTHORITY\SRV_M$

Same issue can happen in AlwaysOn availability group as well if account is set to LocalSystem.

Have you encountered such errors? What was the solution?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Database Mirroring Connection Error 4 – An Error Occurred While Receiving Data: 10054

If you are coming to this page by searching, then I am sure that you are in big trouble. I have come across this error and my SQL Server was not getting started. When I looked into ERRORLOG, I found below errors about the PageAudit Property.

2016-12-30 15:45:36.36 spid9s Starting up database ‘model’.
2016-12-30 15:45:36.37 spid9s Error: 5172, Severity: 16, State: 15.
2016-12-30 15:45:36.37 spid9s The header for file ‘C:\Program Files\Microsoft SQL Server\MSSQL12.DAR_P11D\MSSQL\DATA\model.mdf’ is not a valid database file header. The PageAudit property is incorrect.
2016-12-30 15:45:36.39 spid9s Error: 945, Severity: 14, State: 2.
2016-12-30 15:45:36.39 spid9s Database ‘model’ cannot be opened due to inaccessible files or insufficient memory or disk space. See the SQL Server errorlog for details.
2016-12-30 15:45:36.39 spid9s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

So, since the model database was having serious corruption, SQL was not getting started. The error message essentially means that file header has been corrupted and SQL Server is not able to understand it.

WORKAROUND/SOLUTION

In my case since it was model database, there were two possible options.

1. Get MDF and LDF file from some other server which has “exactly” same version of SQL Server. This would cause to lose data from the previous model database and will get it from another server.
2. If we have a backup of the model and we wish to restore it first, then we need to rebuild system databases using the setup.exe command line. Before rebuilding, we need to keep a copy of MSDB and MASTER database files. As soon as we rebuild, all system database would be as fresh as new installations. After rebuilding, we can restore backups and bring SQL back to life.

After I followed option 1, SQL Server came back online, but SQL Agent was giving trouble. I looked into SQLAgent.out file and here was the information.

2016-12-30 17:04:25 – ? [100] Microsoft SQLServerAgent version 12.0.2269.0 (X64 unicode retail build) : Process ID 83880
2016-12-30 17:04:25 – ? [495] The SQL Server Agent startup service account is NT Service\SQLAgent$SAPSQL.
2016-12-30 17:04:25 – ? [393] Waiting for SQL Server to recover database ‘msdb’…
2016-12-30 17:04:25 – ? [101] SQL Server version 12.00.2269 (0 connection limit)
2016-12-30 17:04:25 – ? [102] SQL Server ODBC driver version 11.00.2100
2016-12-30 17:04:25 – ? [103] NetLib being used by driver is DBNETLIB; Local host server is CLD-SQL01\MIMECAST_MSO
2016-12-30 17:04:25 – ? [310] 8 processor(s) and 17970 MB RAM detected
2016-12-30 17:04:25 – ? [339] Local computer is CLD-SQL01 running Windows NT 6.2 (9200)
2016-12-30 17:04:25 – ! [000] SQLServerAgent cannot start because the msdb database is not available for normal access
2016-12-30 17:04:25 – ? [000] Configuration option ‘show advanced options’ changed from 0 to 1. Run the RECONFIGURE statement to install. [SQLSTATE 01000] (Message 15457) Configuration option ‘Agent XPs’ changed from 1 to 0. Run the RECONFIGURE statement to install. [SQLSTATE 01000] (Message 15457) Configuration option ‘show advanced options’ changed from 1 to 0. Run the RECONFIGURE statement to install. [SQLSTATE 01000] (Message 15457)
2016-12-30 17:04:25 – ? [098] SQLServerAgent terminated (normally)

If you read through above lines, it is easy to identify that below is the cause:

SQLServerAgent cannot start because the msdb database is not available for normal access

When I connected to SQL Server, I found that MSDB database was in suspect state. So, I need to restore that from the backup.

If you ever face error mentioned in a blog title, restore from backup is the only option left. If you find some other solution, please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – The header for file Path is not a valid database file header. The PageAudit property is incorrect

I was trying to help my client in generating a report large data set. After spending some time and understanding the schema, I could provide them the query to get the results. Now, he wanted to save the results in excel sheet. So, he ran the query in SQL Server Management Studio (SSMS), got a lot of rows as output and he hit Ctrl+C in the grid. Let us learn about System.OutOfMemoryException error.

Here is the text of the message (copied using the Copy icon at the bottom left of the message windows)

Exception of type ‘System.OutOfMemoryException’ was thrown. (mscorlib)

If we click on the technical detail icon, we can see below.

Program Location:
at System.Number.FormatInt32(Int32 value, String format, NumberFormatInfo info)
at System.Int32.ToString(String format, IFormatProvider provider)
at System.DateTimeFormat.FormatCustomized(DateTime dateTime, String format, DateTimeFormatInfo dtfi, TimeSpan offset)
at System.DateTimeFormat.Format(DateTime dateTime, String format, DateTimeFormatInfo dtfi, TimeSpan offset)
at System.DateTimeFormat.Format(DateTime dateTime, String format, DateTimeFormatInfo dtfi)
at Microsoft.SqlServer.Management.UI.Grid.StorageViewBase.GetCellDataAsString(Int64 iRow, Int32 iCol)
at Microsoft.SqlServer.Management.QueryExecution.QEResultSet.GetCellDataAsString(Int64 iRow, Int32 iCol)
at Microsoft.SqlServer.Management.UI.VSIntegration.Editors.GridResultsGrid.GetTextBasedColumnStringForClipboardText(Int64 rowIndex, Int32 colIndex)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetClipboardTextForCells(Int64 nStartRow, Int64 nEndRow, Int32 nStartCol, Int32 nEndCol)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetClipboardTextForSelectionBlock(Int32 nBlockNum)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetDataObjectInternal(Boolean bOnlyCurrentSelBlock)
at Microsoft.SqlServer.Management.UI.Grid.GridControl.GetDataObject(Boolean bOnlyCurrentSelBlock)
at Microsoft.SqlServer.Management.UI.VSIntegration.Editors.GridResultsTabPageBase.OnCopyWithHeaders(Object sender, EventArgs a)

Based on my understanding, we read stack from bottom to top. So, if I build a stack by ignoring parameters, it would be like below.

FormatInt32
ToString
FormatCustomized
Format
Format
GetCellDataAsString
GetCellDataAsString
GetTextBasedColumnStringForClipboardText
GetClipboardTextForCells
GetClipboardTextForSelectionBlock
GetDataObjectInternal
GetDataObject
OnCopyWithHeaders

As we can see “Clipboard” – I would assume that its due to copy we are seeing out of memory because we are copying many rows from grid.

WORKAROUND/SOLUTION

As we discovered above, I explained to them that SQL Server Management Studio is not design to handle such kind of requirement. If we want to save the result set into the file, we should save the query output directly to file rather than grid or text in SSMS (and then doing Ctrl + C and Ctrl + V).

Other option would be to follow steps given in one of my earlier blogs

SQL SERVER – Automatically Store Results of Query to File with sqlcmd

SQL SERVER – SSMS Trick – Generating CSV file using Management Studio

I also found that if result is very large, even query execution can fill SSMS buffer and raise same error.

Hope you would be able to work around the issue by using this blog.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER Management Studio – Exception of type ‘System.OutOfMemoryException’ was thrown. (mscorlib)

While applying patches on SQL server standalone instance, in the cluster, it failed with error as below in Summary.txt file. Let us learn about how to fix Exception data is: System.IO.IOException.

Overall summary:
Final result: The patch installer has failed to update the following instance: RTCLOCAL. To determine the reason for failure, review the log files.
Exit code (Decimal): -2058616824
Start time: 2017-01-02 03:23:07
End time: 2017-01-02 03:39:34
Requested action: Patch

When I looked into Detail.txt and searched for error, here was the place of failure.

‘C:\Windows\system32\hadrres.dll’. Exception data is: System.IO.IOException: The process cannot access the file ‘C:\Windows\system32\hadrres.dll’ because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite)
at Microsoft.SqlServer.Configuration.Cluster.UpdateClusterResourceAction.VersionCopy(String source, String target).

Final Parameter Values

(01) 2017-01-02 03:35:41 Slp: Parameter 0 : SQL Server 2012@RTM@KB3072779
(01) 2017-01-02 03:35:41 Slp: Parameter 1 : 0xE349FEF6
(01) 2017-01-02 03:35:41 Slp: Parameter 2 : 0x60797DC7
(01) 2017-01-02 03:35:41 Slp: Parameter 3 : 0xE02C15DF@1356@8
(01) 2017-01-02 03:35:41 Slp: Parameter 4 : 0x24C2C4E7
(01) 2017-01-02 03:35:41 Slp: Parameter 5 : HADRResourceDLLConfigpatch_HADRResourceDLLWorkflow
(01) 2017-01-02 03:35:41 Slp: Parameter 6 : 0x871FC66E
(01) 2017-01-02 03:35:41 Slp: Parameter 7 : 0x27FE6E93
(01) 2017-01-02 03:35:41 Slp: Parameter 8 : 0x27FE6E93

If I understand the message correctly, SQL setup is trying to copy file “hadrres.dll” from “c:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Binn\” to “C:\Windows\system32”. Due to some reason, we are getting error “The process cannot access the file ‘C:\Windows\system32\hadrres.dll’ because it is being used by another process”

Since the patch was incompletely installed, I found that SQL was not starting.

WORKAROUND/SOLUTION

I had no idea as to why copy is not working so I asked to rename files in C:\Windows\system32 folder and stop any monitoring software related services. After renaming, we could apply the patch and found new file copied via setup. We verified and found that version of the file was higher after installation was complete.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – UpdateHADRResource – Failed to version-copy file – Exception data is: System.IO.IOException

As a part of my independent consulting, I also provide a high level overview of the migration strategy. As a part of this engagement, my client wanted me to provide steps to move AlwaysOn availability group from DomainOld to DomainNew. This was happening because they were acquired by another company and they need to migrate all assets to a new domain.

Topology:

Here are the steps which I told them.

• Remove replica Node-03 from availability group.
• Evict Node-03 from the windows cluster.
• Since Node-03 is completely out of AG, we can rebuild Node-03 in DomainNew
• Create a new windows cluster having just one node Node-03.
• Use scripts to migrate all server level objects – logins, jobs, operators from Node-01 to Node-03.
• Node-02 also needs same steps as Node-03 with one difference that it would be part of the same cluster as Node-03 rather than new cluster.
• Configure database mirroring for user databases from Node-01 to Node-03.
• OUTAGE: failover mirror databases to Node03, break the mirror and point application to Node-03.
• Perform test and test and test.
• Evict Node-01 and decommission the original cluster.
• Rebuild Node-01 in the new domain and add it to the cluster.
• Backup and restore the database from the Node-03 to the two new nodes and implement AG across all three new nodes.

After getting above high level guidance, they executed the project and it went fine.

Hope this step would help others as well about AlwaysOn Availability Group.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Migrating AlwaysOn Availability Group to New AD Domain