Author: Root

Connecting to Azure SQL-server from SAS

This post shows how it is possible to connect to an SQL-server server/database (PaaS) in Windows Azure from SAS.

First of all you need to create a SQL-server database. When you create an SQL-server database it will also create a virtual server.The servername for the virtual server is the “funny” name eg. aoe8kg2q9w.
It is on this server, that you need to open for the different IP that need access to the SQL-server.

As shown below you go to the Servers-tab and click on the server where you want to change the firewall settings.

AzureServers

On the server you go to the Configure-tab and enter the IP-adress that has to access the SQL-server.

IPAccess

It is possible to get different connection strings for your SQL-database on Azure. As shown below you need to choose ‘View SQL Database connection strings’.

ConnStrings

Here you can eg find the connection string for an ODBC connection and use it in your libname statement for SAS.
Now you can connect through SAS. First you make a libname. This has only worked for me when I use one of the new ODBC-drivers for Microsoft SQL-server eg. ODBC Driver 11 for SQL Server.

libname azure odbc NOPROMPT=”Driver=ODBC Driver 11 for SQL Server;Server=tcp:<SERVERNAME>.database.windows.net,1433;Database=<DATABASE>;Uid=<USERNAME>@<SERVERNAME>;
Pwd=<PASSWORD>;Encrypt=yes;Connection Timeout=30;” schema=<SCHEMA>;

Below I have made an example. Where I copy sashelp.class to my Azure database.
I need to create an identical copy of sashelp.class on Azure.

proc sql;
create table azure.class like sashelp.class;
quit;

I need this copy because Azure needs a clustered index on the table. If you don’t have a clustered index on the table, it is not possible to load data into the table.

proc sql noprint;
connect to odbc  (NOPROMPT=”Driver=ODBC Driver 11 for SQL Server;Server=tcp:<SERVERNAME>.database.windows.net,1433;Database=<DATABASE>;Uid=<USERNAME>@<SERVERNAME>;
Pwd=<PASSWORD>;Encrypt=yes;Connection Timeout=30;”);
execute  (
CREATE CLUSTERED INDEX idxName ON [SCHEMA].[class]   (    [Name] ASC   )WITH (STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]  ) by odbc;
disconnect from odbc;
quit;

Now that I have made the clustered index on the table. It is possible to load data into the table.

proc sql;
insert into azure.class
select * from sashelp.class;
quit;

Start folder for SAS

It is possible to set the start folder for SAS in different ways. First of all you can do it in the shortcut to SAS in your operating system. This is done by setting the -SASINITIALFOLDER as shown below.

SASInitialFolder

 

 

 

 

 

 

 

 

 

 

 

 

It is also possible to make changes to your sas9v.cfg file. Where you eg. insert the statement -SASINITIALFOLDER “F:\Projects”

Parallel execution in SAS

The example below shows one way to do parallel execution of SAS sessions. In the example below two SAS sessions is started in parallel and a third is executed depending on the result of the two session.

Gets the path of the main SAS program that is executed.

%macro GetSASProgramPath;
 %qsubstr(%sysget(SAS_EXECFILEPATH),1,%length(%sysget(SAS_EXECFILEPATH))-%length(%sysget(SAS_EXECFILEname))-1);
%mend;

 %let SASProgramPath = %GetSASProgramPath;
%put &SASProgramPath;

Starts the first SAS session. %syslput makes it possible to give the SAS session a macrovariable from the main program. In this case it is the path for the SAS program being executed.
wait=no tells SAS not to wait for the SAS session to end before starting the next SAS session.
%sysrput makes it possible to return a macrovariable from a SAS session to the main SAS program. In this cause it whether or not the SAS session executed with og without errors.

option sascmd = “sas”;
signon remote = OneSes;
%syslput _SASProgramPath = &SASProgramPath;
rsubmit process = OneSes wait=no;
 %include “&_SASProgramPath\<SOME PROGRAM.SAS>”;
 %sysrput FirstSessionError = &syscc;
endrsubmit;

Starts the second SAS session.

option sascmd = “sas”;
signon remote = TwoSes;
%syslput _SASProgramPath = &SASProgramPath;
rsubmit process = TwoSes wait=no;
%include “&_SASProgramPath\<SOME PROGRAM.SAS>”;
%sysrput SecondSessionError = &syscc;
endrsubmit;

The statement below waits for the two sessions above to finnish and executes a SAS program if the two sessions did not fail.

waitfor OneSes TwoSes;
signoff OneSes;
signoff TwoSes;
%macro FinalRun;
%if &FirstSessionError < 5 and &SecondSessionError < 5 %then %do;
%put –== No Errors ==–;
%include “&_SASProgramPath\<SOME PROGRAM.SAS>”;
%end;
%mend;
%FinalRun;

Measure performance in SAS

The options fullstimer; gives you a way to do comprehensive performance analysis in SAS. Below is the description of FULLSTIMER from the SAS website.

The SAS System provides the FULLSTIMER option to collect performance statistics on each SAS step, and for the job as a whole and place them in the SAS log. It is important to note that the FULLSTIMER measures only give you a snapshot view of performance at the step and job level. Each SAS port yields different FULLSTIMER statistics based on the host operating system.

Eg.

NOTE: DATA statement used:
real time                   0.06 seconds
user cpu time               0.02 seconds
system cpu time             0.00 seconds
Memory                      88k
Page Faults                  10
Page Reclaims                 0
Page Swaps                    0
Voluntary Context Switches   22
Involuntary Context Switches  0
Block Input Operations       10
Block Output Operations      12

 

Real Time The Real Time represents the elapsed time or “wall clock” time. This is the time spent to execute a job or step. This is the time the user experiences in wait for the job/step to complete. Note: As host system resources are heavily utilized the Real Time can go up significantly – representing a wait for various system resources to become available for the SAS job/step’s usage.
User CPU Time The time spent by the processor to execute user-written code. This is user-written from the perspective of the operating system and not the customer’s language statements. That is all SAS system code that is not operating system code.
System CPU Time The time spent by the processor to execute operating system tasks that support user-written code (all CPU tasks that were not executing user-written code). The user CPU time and system CPU time are mutually exclusive.
Memory Memory represents the amount of memory allocated to that job/step. This does not represent the entire amount of memory that the SAS session is consuming, as it does not reflect any SAS overhead activities (SAS manager, etc.).
Page Faults Represents the number of virtual memory page faults that occurred during the job/step. Page Faults are pages that required an I/O to retrieve (a read was done to the I/O subsystem).
Page Reclaims Represents the number of pages retrieved from the page list awaiting re-allocation (all done in memory). These pages did not require I/O activity to obtain.
Page Swaps  The number of times a process was swapped out of main memory.
Voluntary Context Switches Represents the number of times a process releases its CPU time-slice voluntarily before it’s time-slice allocation is expired. This usually occurs when the process needs an external resource, like making an I/O call for more data.
Involuntary Context Switches The number of times a process releases its CPU time-slice involuntarily. This usually happens when its CPU time-slice has expired before the task was finished, or a higher priority task takes its time-slice away.
Block Input Operations  The number of “bufsize” reads that occur. These are I/O operations to read the data into memory for usage. Not all reads have to utilize an I/O operation since the page being requested may still be cached in memory from previous reads.
Block Output Operations  This represents the number of “bufsize” writes that occur. These are the same as block input operations except that they pertain to the writes to files. As in the case of block input operations, not all block outputs will cause an I/O operation. Some files may still be cached in memory.

Inserting quoted string in where sentence

There are different possibilities when you need to quote a string in SAS.

Before you read any further! Instead of using ‘ ‘ around the macro-variable use “ “.

The SQL-statement below will not work.

proc sql noprint;
insert into ToLib.DestDataset
select *
from FromLib.SourceDataset
where Year = ‘&Year.’;
quit;

The SQL-statement below will work.

proc sql noprint;
insert into ToLib.DestDataset
select *
from FromLib.SourceDataset
where Year = “&Year.”;
quit;

If you have a macro variable containing a year that could be 2012, and you need to put ‘ ‘ or “ “ around it, you can use the macro-functions %bquote as shown below.

My program works like this. I retrieve a year variable into a macro-variable from a configuration dataset.

proc sql noprint;
select upcase(Value) into :Year
from Config.Config_data
where upcase(Name) = upcase(‘Year’);
quit;

I remove blanks (if any)

%let Year = %trim(&Year.);

When I tried to quote the macro-variabel Year (it’s a string not a numeric) to get Year = ‘2012’ I got and error

proc sql noprint;
insert into ToLib.DestDataset
select *
from FromLib.SourceDataset
where Year = %bquote(‘&Year.’);
quit;

I found out that I needed to trim the %bquote-statement again. When I did that it worked.

proc sql noprint;
insert into ToLib.DestDataset
select *
from FromLib.SourceDataset
where Year = %trim(%bquote(‘&Year.’));
quit;

Another way to solve this would be to use

proc sql noprint;
select upcase(Value) format=$quote40. into :Year
from Config.Config_data
where upcase(Name) = upcase(‘Year’);
quit;

This will put “” around the year like “2012”.

Setting up Microsoft BI environment

Great tutorial for setting up a Microsoft BI environment with Microsoft Sharepoint 2013 and Microsoft SQL-server 2012 Enterprise Edition. A lot of the BI features in the Microsoft platform requires an Enterprise Edition of Microsoft SQL-Server 2012.

I have also made a PDF of the tutorial.

Encryption and decryption i C#

This link has by far the most well documented guide to encryption and decryption in C#. It uses the Rijndael Class.

There are three different enryption class for AES in the .NET framework. AES is a standard of Rijndael.

When I get some more time I will write a bit more about encryption in the .NET Framework. I can recommend Pluralsights course on encryption in .NET.

Find partitioned tables in Microsoft SQL-server

This will find partitioned tables

select distinct object_name(object_id) from sys.partitions
group by object_id, index_id
having COUNT(*) > 1

This will find the partitions for a specific table

SELECT *
FROM sys.partitions
WHERE OBJECT_ID = OBJECT_ID(‘<Schema>.<Table>’)