Sunday, November 22, 2015

Exporting Data/Resultset of a Query to CSV

when you need to export the resultset of a query to a csv copying from a resultset window of ssms will give you memory exception when the data is large
Better option will be to use SQLCMD

Also BulkCopy doesnt support queries with multiple lines, but SQLCMD allows you to pass a file containing the query with multiple lines.

Use the below code and paste in notepad and save it as bat file and replace whatever is in red with your parameters.

@ECHO Using SQLCMD
sqlcmd queryout "filepathWithFilename" -c -t , -S DBServerInstance -d databaseName -U username -P password
-i "QueryFileName"
pause

Example:


@ECHO Using SQLCMD
sqlcmd queryout "f:\Output.csv" -c -t , -S ISQL99999q,77777 -d Shipping -U astro -P astrospassword -i "f:\a.txt"
pause
Save it as a bat file

Note: Here the Database name is Shipping 
ISQL99999q,77777 is DB instance Name astro is the username astrospassword is the password


Contents of a.txt file will be the query:
Select * from Shipping.Product  inner Join Shipping.Orders  on Shipping.Orders.ProductId=Shipping.Product.ProductId

Save the a.txt file / the query file and give the same path in the bat file.
 and Run the bat file. 

exporting large data from Database /MS SQL /SSMS to CSV

When there is a need to export lumpsome data from your Database to CSV, the export option wont help you out when you have a custom query or bulk data to export.

You can always use BCP tool for bulk copying.

It is real simple to use , I use it in a batch file to get export several csv's at once.


Use the below code and paste in notepad and save it as bat file and replace whatever is in red with your parameters.

@ECHO Doing Something
bcp "query" queryout "filepathWithFilename" -c -t , -S DBServerInstance -U username -P password

pause

Example:

@ECHO Exporting Orders Table
bcp "Select * from Shipping.Product  inner Join Shipping.Orders  on Shipping.Orders.ProductId=Shipping.Product.ProductId" queryout "d:\Orders.csv" -c -t , -S ISQL99999q,77777  -U astro -P astrospassword

pause


Note: Here the Database name is Shipping and the tables Orders, Product  joined on ProductId
ISQL99999q,77777 is DB instance Name astro is the username astrospassword is the password

Save it as a bat file and Run. 

The output of the file will be the common rows of Table orders and Products saved into D drive with the name Orders.csv


Keep in mind: 
You may see that I have used the database name everywhere ie. on the table names and on join condition. This is the way you are supposed to do it. In case you are having schemas then
DatabaseName.SchemaName.TableName.ColumnName - this should be the format.

Also this doesn't work if your query is large/multiline, for which you can refer the post below.
http://codestruggles.blogspot.in/2015/11/exporting-dataresultset-of-query-to-csv.html

SQL keyboard shorcuts for Frequently used queries

If you work extensively with MS SQL SSMS, you would be looking for a way to reuse the queries instead of typing it all over again. Something like a shortcut for the frequently used queries.

I Use the following ways and methods to access the SQL queries that I frequently use

In SSMS
Setting keyboard shortcuts in SSMS.

Tools > Option > Environment > KeyBoard
There are 11 available shortcuts u can set.

To execute, in the query window
for example the table Orders in the DB NorthWind
you need to just write "Orders" in your query window and select the word and hit Ctrl+0 
As you can see in the screenshot above, it will exec the statement
"Select * from Orders" and get you all the records in your DB.


Unfortunately We are not able to add more shortcuts to SSMS. after lil googling , I found this cool tool called Clavier+

where you can store all your frequently used queries


The other Feature of this is that, you can launch any program with a shortcut, also you can use it to launch any file that you frequently open.

Download:
http://utilfr42.free.fr/util/Clavier.php?sLang=en

Note: Suppose you set a shortcut for keys such as Ctrl+C , V then the action that you specify here will take place instead of the OS's Copy Paste function.

Sunday, September 27, 2015

SQL Timeout Exception Prevention and Tuning Stored Procedures


Prevent Stored Procedure Timeout

Write efficient Stored Procedures

Prevent More Disk Reads

Prevent Dead Locks


System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.


The following are the collection of my learning as a SQL developer.

Because of large data on the tables, the join operations and other operations will take up lot of time hence causing the application to time out, also reducing the performance.


Parameter Sniffing
In short, parameter sniffing is one exec plan for all parameter combinations. 

- This option causes the compilation process to ignore the value for the specified variable and use the specified value instead.
- Unlike the other workaround (Option(Recompile)), this option avoid the overhead of recompiling the query every time.


This is the most common problem, The workaround is to declare the input params  again with different variable name and use the
 new variable through out the SP.

In the Example @local_SetupId

CREATE PROCEDURE [dbo].[pr_get_user_ids]
(
@in_SetupId int
)
As
begin
Declare@local_SetupIdint=@in_SetupId
Select UserIdfrom UserDetailwhere SetupId=@local_SetupId

end




Use Local Temp Tables




Store the data from large tables such as Product / Orders/ Hierarchyinto Local temp Tables and Use them when joining – This is will Prevent the joining of heavy tables.

This is Subject to temp Db Size. Consider the temp Db size.

Select only the needed columns for your Transaction operation and put it into a local temp  table. Hence on performing joins the tables will be lighter  and preventing the use of excess Resource.

In the Example We have filtered out few columns  from productHierarchytable using setupid in Where  clause and used it to join with Product table.

SELECT * into #TempProductHierarchyFROM
(SELECT ProductHierarchy,ProductHierarchyId,ProductIdfromProductHierarchyDetails where SetupId=@local_SetupId)ASTPH          

SELECT ProductCodeInterfacefrommasters.ProductP
inner join #tempProductHierarchyonP.ProductId=#TempProductHierarchy.ProductId

Instead of
SELECT ProductCodeInterface frommasters.Product P
inner join masters.ProductHierarchyDetail PHD onP.ProductId=PHD.ProductId
where P.SetupId=@local_SetupId

Use NON Clustered Index
Use Non Clustered Index on temp tables for quicker Execution

Once you have created a temp table, set index on the primary key of the table – on which you will be performing the join operation.
In the example we have read the Product Hierarchy  table into a temp table and we have created a non-clustered  index onto the primary key i.e ProductId

 SELECT*into#TempProductHierarchyFROM
(SELECT ProductHierarchy,ProductHierarchyId,ProductIdfromProductHierarchyDetails
where SetupId=@local_SetupId)ASTPH                     

Create NonClusteredIndexIDX_TP4 On#TempProductHierarchy(ProductId)



Use With(NOLOCK)
NOLOCK typically (depending on your DB engine) means give me your data, and I don't care what state it is in, and don't bother holding it still while you read from it. It is all at once faster, less resource-intensive

Use NOLOCK on master tables and not everywhere.
Using NoLockmay become dangerous sometimes, so check the query exec plan when in doubt.

Select * from  ProductHierarchy  WITH(NOLOCK)











Join On the InputParameter
This will reduce the operation size and the disk read

Join on input parameter instead of filtering in where condition.
As you can see here we have joined on  setupid i.e the input parameter instead of  joining on the setupid of the other table  and then filtering in where condition. 

SELECT * from trans.PromotionP
inner join masters.ProductPP with(NOLOCK)   on PP.SetupId=@local_SetupId
inner join masters.CustomerCC with(NOLOCK)onCC.SetupId=@local_SetupId

Instead Of
SELECT * from trans.PromotionP
inner join masters.ProductPP with(NOLOCK)   on PP.SetupId=P.SetupId
inner join masters.CustomerCC with(NOLOCK)onCC.SetupId=P.SetupId
Where PP.SetupId=@local_SetupId


Prevent the use of “Select *”

Causes Indexing issues and Binding issues

Dont use "SELECT *" ina SQLquery

Insead use Select ProductId, ProductName from Products



Use EXISTS

To check if any data exists in a particular table, use EXISTS  instead of relying on Count its more effective.


SELECT OrderId,AmendVersionFROMtrans.OrdersWHEREEXISTS(SELECTtop 1 OrderId FROMtrans.OrderExtract WHERESetupId=1099)



Use Local Temp Tables (#TempTableName)







Prevent using Global hash tables (##)


SELECT * into #TempProductFROM(SELECTProductId,LevelId,ProductCodeInterface,NamefromProduct whereSetupId=@local_SetupIdandIsActive=1)ASTP                                          








Use Local Variables to Store the FunctionCall return

Prevent calling the string functions or date functions Over and over again, instead store ‘em in local variables  if you are going to reuse the value.

Declare@sampleStringvarchar(max)='nevermind the promotions'
Declare  @sizeOfStringint
set @sizeOfString=len(@sampleString)
SELECT @sizeOfString




Use Try - Catch

BEGIN TRY
-- Logic / Query here
END TRY           
            
BEGIN CATCH           
 -----------------------------------------------------------------------           
           
  DECLARE           
      @ErrMsg  VARCHAR(255)-- Error Message           
     ,@ErrNo   INT    -- Error Number           
     ,@ErrSeverityINT    -- Error Severity           
     ,@ErrProc  VARCHAR(255)-- Error Procedure           
     ,@ErrLine  INT    -- Error Line           
               
  SELECT           
      @ErrMsg  = ERROR_MESSAGE()           
     ,@ErrNo   = ERROR_NUMBER()           
     ,@ErrSeverity=17           
     ,@ErrProc  = ERROR_PROCEDURE()           
     ,@ErrLine  = ERROR_LINE()           
                           
  RAISERROR  (           
      @ErrMsg           
     ,@ErrSeverity           
     ,1           
     ,@ErrNo           
     ,@ErrLine           
     ,@ErrProc           
     )           
 -----------------------------------------------------------------------           
 END CATCH           
END           
           
           
           
     
 Use  SET NOCOUNT ON

Whenever we write any procedure and execute it a message appears in message window that shows number of rows affected with the statement written in the procedure.

When SET NOCOUNT is ON, the count is not returned.

SETNOCOUNTON
Select PromotionId from Promotion 


Prevent Usage of DDL Statements

Do not try to use DDL statements inside a stored procedure that will reduces the chance to reuse the execution plan.
DDL statements like CREATE,ALTER,DROP,TRUNCATE etc.


Use Alias



If an alias is not present, the engine must resolve which tables own the specified columns. A short alias is parsed more quickly than a long table name or alias. If possible, reduce the alias to a single letter

--Wrong Statement
SELECT PromotionId,P.VersionedPromotionId,Name,PIE.InvestmentTypeIdfromPromotion P
Inner join PromotionInvestmentPIE onPIE.VersionedPromotionId=P.VersionedPromotionId
where P.Name='Blah'

--Correct Statement
SELECT P.PromotionId,P.VersionedPromotionId,P.Name,PIE.InvestmentTypeIdfromPromotion P
Inner join PromotionInvestmentPIE onPIE.VersionedPromotionId=P.VersionedPromotionId
where P.Name='Blah'



Don't use UPDATE instead of CASE

Take this scenario, for instance: You're inserting data into a temp table and need it to display a certain value if another value exists. Maybe you're pulling from the Customer table and you want anyone with more than $100,000 in orders to be labeled as "Preferred." Thus, you insert the data into the table and run an UPDATE statement to set the CustomerRankcolumn to "Preferred" for anyone who has more than $100,000 in orders. The problem is that the UPDATE statement is logged, which means it has to write twice for every single write to the table. The way around this, of course, is to use an inline CASE statement in the SQL query itself. This tests every row for the order amount condition and sets the "Preferred" label before it's written to the table.






Avoid Functions on RHS

Dont use this

select *
from Promotion
where YEAR(StartDate)=2015
and  MONTH(StartDate)=

Use this

Select *
From Promotion
Where StartDatebetween'6/1/2015'
    and'6/30/2015'



 Specify optimizer hints in SELECT

most cases the query optimizer will pick the appropriate index for a particular table based on statistics, sometimes it is better to specify the index name in your SELECT query.
Do not use this unless you know what you are doing.

SELECT *
FROM Promotion
WITH ( Index(IdxPromotionId))
WHERE Name ='blah'
and Setupid=1099 




Hope these tips will help you prevent and solve the timeout exception  you face. If you want to add any please mention in the comments.