The program that I'm going to show you simulates a drug study with three groups (Placebo, Drug A, and Drug B), two genders, and correlated values of systolic blood pressure (SBP) and diastolic blood pressure (DBP).

proc format; ❶
   value $Gender '0' = 'Male'
                 '1' = 'Female';
run;
 
data Study;
   call streaminit(13579); ❷
   do i = 1 to 30;
      do Drug = 'Placebo','Drug A','Drug B'; ❸
         Subjec + 1; ❹
         Gender = put(rand('Bernoulli',.5),1.); ❺
         SBP = rand('Normal',160,10) - 20*(Drug = 'Drug A') ❻
            - 30*(Drug = 'Drug B') - 5*(Gender = '1');
         DBP = .5*SBP + 40*rand('Uniform'); ❼
 
         SBP = round(SBP); ❽
         DBP = round(DBP);
         output;
      end;
   end;
   drop i;
   format Gender $Gender.;
run;

❶ Create a character format that will be used to format Gender.
❷ Because this is a drug study, you want to be able to reproduce the same random sequence at a later time. If you use the same argument in CALL STREAMINIT in another program, you will create identical random streams.
❸ Remember that SAS DO LOOPS can use character values. This is very handy.
❹ This SUM statement is generating the variable Subject.
❺ The Bernoulli distribution with the second argument set to .5, will assign gender (0 or 1) with a probability of .5 for both. The PUT function in this statement is doing a numeric-to-character conversion.
❻ This is a very interesting statement. You start by choosing values for SBP from a normal distribution with a mean of 160 and a standard deviation of 10. The logical expressions such as (Drug = 'Drug A') returns a 1 if the expression if true and 0 otherwise. By multiplying this by a numeric value, you can adjust the means in the three groups. The Placebo group will have a mean close to 160—the subjects in drug group A will be approximately 20 points lower than the Placebo group and the subjects in drug group B will be approximately 30 points lower than the Placebo group. Finally, females will be approximately 5 points lower than males.
❼ DBP will be correlated with SBP because it includes a proportion of SBP in the calculation. (See the previous simulated data blog for more details about generating correlated pairs.)
❽ Although I could have added the ROUND function when SBP and DBP were created, it makes the program a bit easier to read by rounding the values in separate statements.
The figure below shows the first nine observations in the simulated Study data set. This figure as well as all the box plots and the scatter plot that follow, were created using built-in SAS Studio tasks in the cloud version called SAS OnDemand for Academics.

The next figure is a box plot with SBP as the displayed variable. As a quick review, a box plot shows a vertical line at the median (in the middle of the box). The lower and upper edges of the box represent values at the 25^th and 75^th percentile respectively. The diamond in the box represents the mean. The distance between the 25^th and 75^th percentiles is called the inter-quartile range. Finally, any points more than 1.5 inter-quartile ranges below the 25^th percentile or above the 75^th percentile are plotted as circles and considered outliers. You can see one outlier in the placebo group in the left portion of the box plot below.

Notice that the means are approximately where you expect them to be (Placebo around 160, Drug A around 140, and Drug B around 130. The means are slightly lower because half of the subjects are female and their systolic blood pressures were designed to be approximately 5 points lower than the males.

Here is a plot of SBP by Gender.

The last figure is a plot of SBP by DBP. Notice that the values are correlated (the Pearson r is approximately .64).

Using techniques similar to this program, combined with methods in the previous simulation blog, you should be able to create your own custom simulated data sets.

Creating Simulated Data Sets (Part Two) was published on SAS Users.

Certainly, there will be commonalities – for example, all people need time to sleep, to eat and many people need time to work and study. Also, for sure there are differences in how people divide their time for activities in each day. There might be some pattern of time use in different countries and different cultures. I am interested in exploring this, so I found some data from the web to explore.

What is a Time Use Survey?

Over the last 30 years, an increasing number of countries around the world have been conducting large-scale time use surveys. The Time Use Survey is designed to measure the amount of time people spend on various activities in their daily life, across a total duration of 24 hours (or 1,440 minutes). These activities, such as work, relaxing, and exercising, are classified into a set of descriptive categories, and the time on these activities are interviewed from some respondents. Then the data was recorded, calculated and edited.

I got the time use data from OECD (Organization for Economic Co-operation and Development) site, and the time use survey was conducted in more than 30 countries from 2009 to 2016. I also got the American Time Use Survey data for 2020 for my exploration. I am aware that the data quality might not be good enough for serious research, but that’s not a problem for me. I just want to explore it for fun, while practicing SAS Visual Analytics usage.

How do people around the world spend their time?

Download the excel file from OECD site, import it in SAS Visual Analytics. I will explore how people in different countries spend their time, how many minutes do they averagely spend on among the five categories (according to OECD, they put different activities into five categories).

We can easily draw a bar chart in SAS VA like below. Note: the downloaded OECD data has the time use data for American, but I eliminate it from this chart due to its total is 1,469 minutes (more than 24 hours a day). And that leads me to explore the American time use data separately.

See the green bars are the longest one among the five colored bars? They represent Personal Care. It seems people across these countries pay the most time in Personal Care. Unbelievable? Check the activities in the Personal Care category: sleeping, eating, dressing, and others personal care activities. All right, people sleep about 8 hours (480 mins) every day on average, that’s about 30% of a day. It makes sense that the Personal Care category occupies the most time (about 661 mins on average) in our daily life.

Now from another perspective, let’s see the top and bottom countries where people spend time on Personal Care, as well as on paid work/study. From below charts, I guess you won’t be surprised when seeing France sits on the top one country with most Personal Care time, and Japan sits on the top one country with most time on paid work/study, while Italy is the country where people spend least time on paid work/study.

Note in above charts, I use the same scale for the X axis intentionally. This is to make sure people get direct feel on the differences between the two categories, the ‘Paid work or study’ time on the right is less than half of the ‘Personal Care’ time on the left.

Furthermore, we can look at the distribution of these five categories across all these countries. Calculate the percentage for each major category using calculated items in VA and show them in a box plot. We see people on average spend about 46% of their time on Personal Care, about 20% on Leisure, and 19% on Paid work/study. The highest percentage in ‘Personal Care’ is about 52%, more than 12 hours every day. The least percentage is about 42%, that’s about 10 hours every day. Also notice that time on Personal Care, Leisure and Paid work/study are the top 3 categories that takes more than 85% time each day.

How do Americans spend their time?

As I mentioned, the American data from OECD is not ideal for me, so I downloaded the American Time Use Survey (ATUS) data, and using the 2020 data file for further exploration. The ATUS data was organized in different categories using different methodology than OECD data, so I must do data preparation in SAS Studio, and then explore in SAS Visual Analytics.

Prepare the data

The raw 2020 data file has 399 columns, and 8,782 rows. It contains data for the total number of minutes that each respondent spent on the 6-digit activity (per ATUS code rule). The column names with the letter "t" precedes the 6-digit codes, identify activities by their 6-digit codes. The first and second digits of the 6-digit code correspond to some tier1 code; the second and third digits correspond to some tier 2 code, etc. Each row corresponds to a unique respondent.

So my data preparation includes:

• Classify the 6-digit activities to their corresponding tier1 codes, which comes to about 18 categories.
• Calculate the means and 99% confidence interval for each of the 18 categories.
• Transpose the dataset and merge the datasets. If you are interested in how I did this, you can get the code on GitHub..
• The ATUS data set contains one column on Age, so I can make a custom category of age group in VA and divide the ages to three categories: less than 18, great than 65, and between 18 and 65. This will enable me to compare the ATUS data with the OECD data (whose ages are between 18 and 65).

Aggregate the data

ATUS contains detailed data from thousands of respondents with hundreds of columns. I need to aggregate the data for my exploration. Here are some tips when doing the aggregation for each of these hundreds of columns:

1. The default aggregation for measure items in VA is Sum. We can easily change the aggregation in the data pane by clicking the ‘Edit properties’ icon and choosing other aggregation (I will use ‘Average’) in the ‘Aggregation’ dropdown list. But when I have hundreds of measure items in the ATUS data set, how can I quickly set the average aggregation for them instead of one-by-one? The tip is clicking on the first measure item, and scroll to the last item, press ‘shift’ when clicking the last item. This will select all these measure items. Right click the mouse, and from the pop-up menu, choose Aggregation > Average. This will set the aggregation to average for all the chosen items.
2. I need a bunch of calculated items; each comprise lots of measure items. In SAS Visual Analytics, we can manually add each item in visual mode. But it’s too tedious to add so many measure items. The tip here is to write some SAS macro codes to generate the calculation expressions in text for me, then copy/paste the expression in text mode.

Explore the data

According to the ATUS code rule, ATUS uses different categories than OECD categories. To be able to compare the time used in major activity categories, I make the similar major activity categories like those from OECD, based on my personal understanding of the ATUS activities. Then with the bunch of calculated items, I get the time for these major activity categories. Due to methodology difference, be aware that this may lead the results to be partially inaccurate.

Now starts my ATUS exploration. Below charts show how people in American divide their daily time. The dataset has information on gender, so the bottom one shows the average percentage for Male and Female respectively.

When I put the percentage data (calculated for major activities categories) in a box plot, it has lot of outliers for each category. Considering different methodology and personal classification to the major activity categories (here is the OECD code), I see some difference than the OECD box plot. Note that the ranking for top two major activity categories are Personal Care and Leisure, the same pattern as in OECD data.

Identify the outliers

Notice those outliers in above box plot? I’d better explore more. In latest version, SAS Visual Analytics will automatically detect outliers in data items. This ‘Insights’ will list the data items in report objects that might be affected by outliers.

For example, in below screenshot, I made a histogram of the ‘Personal Care %’, which shows its distribution looks like normal. If I click the ‘Insight’ icon at top-right corner, VA will show all the data items that might be affected by outliers. If clicking the icon next to the ‘Personal Care %’ item at the bottom, a message will pop up saying that there are 243 outliers in this data item.

Create a custom graph

I saw lot of outliers in columns of ATUS data when exploring it, so I decide to use the mean value with confidence intervals. I created a custom graph with a scatter plot and a schedule chart. In SAS VA, assign the black dot in the custom graph to show the mean value and make the beginning and ending of each blue bar show the 99% confidence intervals.

Below is the top 10 ATUS activity categories (here are the ATUS tier 1 code categories) American people spend time on. We see the largest average time is the Personal Care, about 586 mins (nearly 10 hours) with 99% confidence intervals ranging from 583.5 to 588.4 mins.

That’s my initial exploration of Time Use Survey data, but much more can be done. For example, because ATUS data is collected on an ongoing, monthly basis, we can perform time-series analysis to identify changes in how people spend their time.

Would you like to give it a try? Visit the SAS Visual Analytics Gallery on the SAS Support Communities to see more ways you can use SAS Visual Analytics to explore data. Then sign up for a two-week free trial of SAS Visual Analytics.

How do people divide their time among daily activities? was published on SAS Users.

So, first, let’s define the purpose of my challenge:

deploy a basic calculator API service capable of executing the following operations for two input decimal numbers: addition, subtraction, multiplication, and division

The challenge must be executed under these restrictions:

• Usage of a machine learning model as the “compute-engine” of the calculator
• Development under the low code / no code paradigm
• The end-to-end setup and execution process should not take more than a couple of hours

Use the following tools in SAS Viya:

• Model Studio
• Intelligent Decisioning
• Simple web app (set up not covered)

The plan

The steps that I am identifying to complete my challenge are:
Step 1 - Create a machine learning model representing the compute-engine of my calculator (Model Studio)
Step 2 - Determine how to process other mathematical operations
Step 3 - Step 3 - Embed the needed logic into the decision to perform the calculator operations (Intelligent Decisioning)
Step 4 - Publish the artifact created as an API service (web app created outside of this post)

Step 1. Create a machine learning model as the compute-engine

Our first step is to create a model. We start with the addition operation and build from there. We’ll perform the addition by normal means of adding two numbers. Next, we’ll apply some extra logic which will perform subtraction, multiplication, and division. The diagram below represents the process:

A machine learning model is built from a data set where it self-learns what to do. I want my model to learn the addition of two numbers. I created a training data set in Excel with 100 registers, each of them with two random numbers between 0 and 1 and then the sum of them. The image below displays the general setup:

The algorithm / model I chose for my compute engine is the linear regression. Linear regression is a simple machine learning algorithm based in the following formula:

y = a + b·X₁ + c·X₂ + d·X₃ + … + n·X_n

Where:

• y is the result of the model execution – the result of the addition operation
• X₁, X₂, X₃, …, X_n are the input variables – for the calculator, there only will be X₁ and X₂ as operands
• a, b, c, d, …, n are the parameters the machine learning process determines to create the model

So, with the training data set created, I’ll open a new machine learning project in SAS Viya Model Studio, selecting my data set from where the algorithm will learn, assign the target variable, add linear regression node, a test node, and click “Run pipeline”. Note: if following along in your own environment, make sure to use Selection Method = Adaptive LASSO and toggle Suppress intercept = On in the linear regression node. The resulting model resembles the following:

You can learn more about model creation in the How to Deploy Models in SAS tutorial on the SAS Users YouTube channel.

Once the pipeline completes, and reviewing the initial results, it seems the model is behaving in a proper way; but when I test specific operands where the result is zero, I realize the model has misgivings:

Those are the operations with zero as result. I think that maybe, the algorithm hasn’t learned with the proper data, so I change the first seven registers of my initial dataset with the following operations with zeros:

Again, running the pipeline and letting the magic work, Voila!!!, the process has learned to handle the zeroes and properly sum two input numbers. When I check the results, I verify the results calculated (predicted) by the model are the same as the original ones that were indicated in the training and test dataset. So now, I am sure my new model is ready for use as my calculator compute engine.

Now that I have my compute engine (model) ready, it’s time to use it. We know it can perform the sum operation perfectly, so how do we perform the rest of the operations? We’ll take the sum model, move it into SAS Intelligent Decisioning, and add rules to handle the rest of the operations.

First, let’s explore the logic that will build the operations from our original model. This is where mathematics start playing in (Step 2). After exploring the operations we'll look at the Decision model where we'll define the logic to run the operations (Step 3).

Addition

Execute the model with the two input numbers, with no additional logic.

Subtraction

By changing the sign of the second input, the model does the rest.

That’s a simple enough solution for subtraction, but how do we handle multiplication and division? Let’s take a look.

Multiplication and division

How can I perform a multiplication or a division operation if my compute engine only executes the addition operation? Here we can apply the magic of logarithmic properties stating:

• log (x*y) = log(x) + log(y)
• log (x/y) = log(x) – log(y)

Following this logic, if I want to multiply two numbers, I calculate the logarithm of each one and perform the addition operation in my compute engine. I follow this up by applying the exponential function to reverse the logarithm. The image below outlines the workflow.

For the division, it is the same process, but changing the sign of the second logarithm to the subtraction operation.

Additional Logic

There are also certain cases requiring special treatment. For example, a division by zero generates an error, a logarithm of zero or a negative number cannot be calculated, and the multiplication of two numbers is zero if at least one of them is zero.

Let's now build the calculator in SAS Intelligent Decisioning, including the operations, the model, and the extra logic.

Step 3 - Embed the needed logic into the decision to perform the calculator operations

The diagram below represents the Decision flow for our calculator example. Each node is numbered and followed by a definition.

0 - Overall decision flow - definition doc found here on GitHub
1 - Determine if zero is a factor for multiplication or division operations - definition doc found here on GitHub
2 - Decision handles value of previous step - set Variable = Decision_Internal_EndProcess = True
3 - Process calculations based on operation value - definition doc found here on GitHub
4 - Calculator linear regression model created earlier in the post - model definition file found here on GitHub
5 - Additional logic to finalize processing on multiplication and division operations - definition doc found here on GitHub

Step 4 - Publish the final artifact created as an API service

After completing all the work on the decision, click on the Publish button and the Calculator is ready to be consumed via an API.

A colleague of mine created a simple web application which calls models using SAS Viya microservice APIs. I'll use this web app to display the results of my calculator. For brevity, I won't cover the details of the app. If you'd like to see how to create a web app using SAS APIs, I recommend the Build a web application using SAS Compute Server series on the SAS Communities.

The app allows me to choose my decision flow, add my operands and indicate an operation as seen below.

I tested with several operand and operation combinations and they all checked out. It worked!

Final Thoughts

I can consider my self-challenge solved. Through this example we accomplished the following:

• The Calculator API Service can perform the four operations based on a Machine Learning Model.
• I created a simple Machine Learning Model to perform the addition of two decimal numbers from a 100 registers data set.
• The model and the extra logic needed to perform the operations was developed under the low code / no code paradigm.
• I used the visual interface to generate the model and the extra logic, in conjunction with the expression builder, to apply the logarithm and exponential operations.
• The overall process has taken no more than a couple of hours.

Apart from the usefulness of this API, my principal takeaways of this self-challenge are:

• In this case, building my data set to obtain the exact behavior I wanted for my model was quite straight-forward.
• Building the model through the Graphical User Interface was easy and fast.
• Having the capacity to embed the models with extra logic under the low code / no code paradigm provides “supercharged intelligence” to the model
• The publishing feature of the whole artifact as an API service is great, providing instant value to the consumers.
• SAS Viya is a great platform as it provides all the pieces needed to satisfy your analytical business needs as well as your “curiosity needs”.

How I used a SAS ML model and Intelligent Decisioning to build a calculator was published on SAS Users.

Observing all environments

SAS Enterprise Session Monitor aims to be the single point of contact for observing distributed systems. It brings unparalleled visibility to understanding environments using detailed system and application-level metrics for every session of SAS that is launched. This goes beyond traditional monitoring and into observability—aggregating, correlating, and analyzing a steady stream of constant data from systems to effectively troubleshoot or debug environments and sessions. Sessions in this case are those that come from SAS 9, SAS Viya, build servers, testing environments, and more. SAS Enterprise Session Monitor receives that data, displays it live in the tool, and stores the data for historical review in an embedded database.

SAS Enterprise Session Monitor is extensible and customizable: administrators can build patterns using regular expressions to track third-party sessions or custom in-house applications. If a process runs on a Windows or Linux server, SAS Enterprise Session Monitor can be configured to record metrics about it.

What metrics are collected?

SAS Enterprise Session Monitor collects and stores system metrics and logs, many monitoring tools do. Here is where things begin to get interesting, however: SAS Enterprise Session Monitor collects application-level metrics about SAS user sessions. The size of the SASWORK area is monitored and the amount of space in the CAS_DISK_CACHE. Users of SAS Enterprise Session Monitor are able see within DATA and PROC steps as code executes within SAS 9 or SAS Compute Server sessions. SAS Viya users can see the CAS actions that execute within their CAS sessions.

This information is presented in the form of spans which appear on a time-series graph along with session information such as CPU usage, memory usage and disk usage. This user activity is tracked for all user sessions, across all platforms. This code-level analysis can help to understand which SAS Procedures are used, which (and how frequently) datasets are opened, and which users are using the environments at different times.

Grand central admin-station

Administrators use SAS Enterprise Session Monitor to make sure their environments are stable and performant. Historical data can be used to profile workloads, charge back departments or help promote jobs between development, testing and production environments. Critical system resources are tracked to better understand when peak usage time is and to understand where resource constraints occur. This stored historical data can also be used for troubleshooting purposes, and all sessions and jobs can be searched for error events to help in problem analysis. Profiles of scheduled batch jobs can be graphed to see when large numbers of sequential programs could be redesigned to run in parallel. SAS Enterprise Session Monitor knows when distributed workloads should be linked together – in a SAS Grid or MPP CAS deployment.

Lower total cost of ownership

Administrators can use SAS Enterprise Session Monitor to accurately right-size their infrastructure with all the metrics collected — whether that is in the cloud or on premises. Accurate user counts and licensing can be determined for concurrent users in all distributed environments. And with accurate information coming in from distributed environments and multiple nodes, potential problems can be identified, and administrators can accelerate time to resolution and reduce system downtime on production or business-critical systems.

A drag-and-drop interface also allows for workloads from different teams to generate cost allocation rules so that costs can be charged back to departments depending on their usage of system resources. This allows for accurate tracking of cooperative resource sharing.

Empowering development teams

Developers (data scientists, analysts or programmers) use SAS Enterprise Session Monitor in real time to monitor or view progress of their code as it runs. This improves the developer experience and closes the feedback loop as they can see issues before something is promoted to production. Developers can use it to prioritize jobs and have insight into what is happening during their program execution.

This empowers individual programmers, as well as teams of developers: teams can be configured to have access to their other team members’ sessions in SAS Enterprise Session Monitor. Privileged users can also be configured to allow team leads or power users to terminate sessions and view SAS program logs from the SAS Enterprise Session Monitor interface in a secure and audited way.

Other tidbits

I mentioned how SAS Enterprise Session Monitor can analyze batch job flows, visualizing them into graphs that display total runtime and dependencies. Taking this a step further, the batch job flows can be viewed through Relative Comparisons — a feature where two defined time spans can be compared. Simply put, this means that one set of scheduled work can be compared to a previous run. This can give detailed information when evaluating whether to change a program or model, or when performing root-cause analysis of issues that impact the runtime of the scheduled work.

Lastly, developers can use real-time custom chart annotations that show up on the time-series graph. The %esmtag statement generates these annotations and can be used much like %put statements. These can be used as status checkpoints or observation counts, providing feedback in real time as the developer watches the program execute. These annotations are searchable in SAS Enterprise Session Monitor.

Summary

I hope you can feel my excitement about this tool and are able to see a few reasons to check this offering out — the potential for what can be monitored is almost endless. Here’s a quick recap:

• Enterprise Session Monitor provides visibility into many different types of SAS workloads. Servers and microservices across multiple SAS 9 and SAS Viya environments can be monitored in one place. Even third-party tools and data sources can be monitored with a little customization.

• Developers use it to close the feedback loop when developing new SAS programs.

• Administrators use it to solve platform issues—through session management, live data and historical data about SAS processes and system resources.

Additional resources

SAS Enterprise Session Monitor documentation

Configuration and Usage of SAS Enterprise Session Monitor

SAS Enterprise Session Monitor - Obsessing over Observability was published on SAS Users.

Because most programs that create simulated data use random numbers, let's start off by discussing the RAND function. This function can generate random numbers that follow distributions such as uniform, normal, Bernoulli, as well as dozens of other distributions. Veteran SAS programmers might be more familiar with some of the older random number functions such as RANUNI and RANNOR. RANUNI was used to generate uniform random numbers (numbers between 0 and 1) and RANNOR generated random numbers from a normal distribution. The RAND function has replaced all of the older functions and has a number of advantages over the older functions.

The first argument of the RAND function is the name of the distribution that you want to use, such as Uniform, Normal, or Bernoulli. For some of the distributions, such as Normal, you can supply parameters such as the mean and standard deviation. Here are some examples:

Important Note: if you want a reproducible series of random numbers using the RAND function, you must seed it by a call to STREAMINIT (with a positive integer argument) prior to its use. For example:

       call streaminit(132435);

To clarify the note above, here are two programs that use the RAND function—one with, and one without the call to Streaminit.

 data Without;
   do i = 1 to 5;
      x = rand('Uniform');
      output;
   end;   drop i;
run;

Here is the output from running this program twice.

Notice that the values of x are different in each run. Now let's run the same program with CALL STREAMINIT included. Here is the program.

data With;
   call streaminit(13579);
   do i = 1 to 5;
      x = rand('Uniform');
      output;
   end;
   drop i;
run;

And here are the output listings from running this program twice.

Adding CALL STREAMINIT creates the same sequence of random numbers each time the program is run. This is useful if you are generating groups for a drug study and want to be able to re-create the random sequences when it comes time to break the blind and analyze the results. Another reason I sometimes want to generate a repeatable sequence of random numbers is for problem sets included in many of my books—I want the reader to get exactly the same results as I did.

Let's switch topics and see how to write a program where you want to simulate flipping a coin. The program below uses a popular method, but not it is not as elegant as the next program I'm going to show you.

*Old fashioned way to generate "random" events;
data Toss;
   do n = 1 to 10;
      if rand('uniform') lt .5 then Result = 'Tails';
      else Result = 'Heads';
      output;
   end;
run;

In the long run, half of the uniform random numbers will be less than .5, and the proportion of heads and tails will be approximately .5. Here is a listing of data set Toss.

A more sophisticated approach takes advantage of the RAND function's ability to generate random number from multiple distributions. A Bernoulli distribution is similar to a coin toss where you can adjust the probability of a 1 or 0 by including a second parameter to the function. The Toss2 program, shown below, does just that.

*More sophisticated program;
proc format;
   value Heads_Tails 0="Heads" 1="Tails";
run;
 
data Toss2;
   do n = 1 to 10;
      Results = rand('Bernoulli',.5);
      format Results Heads_Tails.;
      output;
   end;
run;

The format Heads_Tails substitutes the labels "Heads" and "Tails" for values of 0 and 1, respectively. Here is a listing of data set Toss2.

The final discussion of this blog, concerns generating random values of two or more variables that are correlated. The example that follows generates x-y pairs that are correlated.

*Creating correlated x-y pairs;
data Corr;
   do i = 1 to 1000;
      x = rand('normal',100,10);
      y = .5*x + rand('Normal',50,10);
      output;
   end;
   drop i;
run;

By including a proportion of the x-value when creating the y-value, the x- and y-values will be correlated. Shown below is the output from PROC CORR, showing that x and y are correlated (r = .45586).

I used a SAS Studio task to create the scatterplot shown next.

You can increase or decrease the correlation by increasing the proportion of x used to create y. For example, you could use y = .8*x + rand('Normal',20,10); to create x-y pairs with a higher correlation.

You can see more examples of the RAND function in my book, SAS Functions by Example, Second Edition, available as an e-book from RedShelf or in print form from Amazon.

To learn more about how to use SAS Studio as part of OnDemand for Academics, to write SAS programs, or to use SAS Studio tasks, please take a look at my new book: Getting Started with SAS Programing: Using SAS Studio in the Cloud (available in e-book from RedShelf or in a paper version from Amazon).

I hope you enjoyed reading this blog and, as usual, I invite comments and/or questions.

Creating Simulated Data Sets was published on SAS Users.

SAS Analytics Pro consists of three core elements of the SAS system: Base SAS®, SAS/GRAPH® and SAS/STAT®. The containerized deployment option adds the full selection of SAS/ACCESS engines making it even easier to work with data from virtually any source.

Even better, the containerized deployment option now adds new statistical capabilities that are not available in SAS/STAT on SAS9. Thanks to SAS Viya’s continuous delivery approach, we are able to provide this additional functionality so soon after the initial release.

Below are highlights of these additional capabilities (you can find more details by following the links):

Causal Inference Procedures

• Carry out causal mediation analysis of time-to-event data (either by Cox proportion hazard model or accelerated failure time model) in PROC CAUSALMED.
• Use new causal effect identification methods, import and export models in PROC CAUSALGRAPH.
• PROC CAUSALGRAPH also supports expanded grammar options and the PATHDIAGRAM statement outputs graphical path diagrams.

Bayesian Analysis Procedures

• Model multinomial data with cumulative probit, cumulative logit, generalized link, or other link functions in PROC BGLIMM.
• Specify fixed scale values in a generalized linear mixed-effects model, and use an improved CMPTMODEL statement in PROC MCMC and PROC NLMIXED to fit compartment models.

Survey Procedures

• Compute new design-adjusted confidence limits for proportions in PROC SURVEYFREQ.
• Use new random assignments options in PROC SURVEYSELECT.
• Request new confidence limits, such as the Agresti-Coul limits in PROC FREQ.

Additional Capabilities:

• The new MARGINS statement in PROC GLIMMIX computes predictive margins of fixed effects.
• The Fast Quantile Profess Regression solver improves performance in PROC QUANTREG. For complete information on all new statistical features that the containerized Analytics Pro provides, see the SAS/STAT documentation.

For those SAS customers already on SAS Viya, or those considering the move, SAS Analytics Pro provides one more example of the new powers you will enjoy!

Additional statistical capabilities in the containerized deployment of SAS Analytics Pro was published on SAS Users.

I recently completed a series of articles on the SAS Community library where I shed some light on different techniques for feeding web applications with SAS data stored in SAS Viya environment.  The series includes a discussion of options for extracting data, building a React application, how to build web applications using SAS Viya, SAS Cloud Analytic Service (CAS), SAS Compute Server, and SAS Micro Analytic Service (MAS).

I demonstrate the functionality and discuss project details in the video Develop Web Application to Extract SAS Data, found on the SAS Users YouTube Channel.

I'm tying everything together in this post as a reference point. I'll provide a link to each article along with a brief description. The Community articles have all the detailed steps for developing the application. I'm excited bring you this information, so let's get started.

Part 1 - Develop web applications series: Options for extracting data

In this first article, I explain when to use SAS Micro Analytic Service, SAS Viya Jobs, SAS Cloud Analytic Service, and SAS Compute Server.

Part 2 - Develop web applications series: Creating the React based application

To demonstrate the different options, in the second article, I create a simple web application using React JavaScript library. The application also handles authentication against SAS Viya. The application is structured in such a way to avoid redundant code and each component has a well-defined role. From here, we can build the different pages to access CAS, MAS, Compute Server or SAS Viya Jobs.

The image below offers a view of the application which starts in Part 2 and continues throughout the series..

Part 3 - Develop web applications series: Build a web application using SAS Viya Jobs

In this article, I drive you through the steps to retrieve data from the SAS environment using SAS Viya Jobs. We build out the Jobs tab and on the page, display two dropdown boxes to select a library and table. The final piece is a submit button to retrieve the data to populate a table.

Part 4 - Develop web applications series: Build a web application using SAS Cloud Analytic Service

In article number 4, we go through the steps to build a page similar to the one in the previous article, but this time the data comes directly from the SAS Cloud Analytic Service (CAS). We reuse the application structure which was created in Part 2. We focus on the CAS tab. As for the SAS Viya Jobs, we display two dropdown boxes to select a library and table. We finish again with a submit button to retrieve the data to populate a table.

Part 5 - Develop web applications series: Build a web application using SAS Compute Server

In the next article, we go through the steps to build a page similar to the ones from previous articles, but this time the data comes directly from the SAS Compute Server. We reuse the application structure created in this Part 2 article. The remainder of the article focuses on the Compute tab. As for the CAS content, we display two dropdown boxes to select a library and table. Finishing off again with the submit button to retrieve the data to populate a table.

Part 6 - Develop web applications series: Build a web application using SAS Micro Analytic Service

For the final article, you discover how to build a page to access data from the SAS Micro Analytic Service. We reuse the same basic web application built in Part 2. However, this time it will require a bit more preparation work as the SAS Micro Analytic Service (MAS) is designed for model scoring.

Bonus Material - SAS Authentication for ReactJS based applications

In this addendum to the series, I outline the authorization code OAuth flow. This is the recommended means of authenticating to SAS Viya and I provide technical background and detailed code.

Conclusion

If you followed along with the different articles in this series, you should now have a fully functional web application for accessing different data source types from SAS Viya. This application is not for use as-is in production. You should, for example add functionality to handle token expiration. You can of course tweak the interface to get the look and feel you prefer.

See all of my SAS Communities articles here.

Creating a React web app using SAS Viya was published on SAS Users.

View more Problem Solvers posts

This article is co-authored with my colleague Russ Tyndall, a Senior Principal Technical Support Analyst at SAS.

As defined in the SAS^® 9.4 Stored Processes: Developer's Guide, Third Edition, a stored process "is a SAS program that is stored on a server and defined in metadata, and which can be executed as requested by client applications." One of the benefits of using stored processes is that client applications always have the latest version of the code. In addition, stored processes provide enhanced security and application integrity.

Have you ever submitted a stored process, and instead of the expected output, you saw errors or no output at all? Depending on how you submit the stored process, various logs are available to assist you with debugging.

This article provides guidance for understanding which situations call for which logs, where to find each log, and what you should look for in each log. The article uses two clients as examples: SAS^® Enterprise Guide^® and the SAS^® 9.4 Stored Process Web Application.

The article is divided into two sections:

• Frequently Used Logs
• Infrequently Used Logs

Frequently Used Logs

The logs that are described in this section are the most prevalent logs that you request when a stored process does not execute properly or when there seems to be a problem with the stored-process server.

SAS^® Object Spawner log

What

The SAS Object Spawner log records anytime that the spawner tries to start or stop a stored-process server, a workspace server, or a pooled workspace server. It also records when a request is redirected to a running stored-process server or a pooled workspace server or to a SAS® Grid Manager server.

The syntax for this log's name is ObjectSpawner_yyyy-mm-dd_machine-name_process-ID.log.

Where

The default locations for the object-spawner log are as follows:

• Microsoft Windows operating environments: SAS-configuration-directory\Lev1\ObjectSpawner\Logs
• UNIX operating environments: SAS-configuration-directory/Lev1/ObjectSpawner/Logs

If the logs are not in the directories that are listed above, you can check the logconfig.xml file that resides in the SAS-configuration-directory\Lev1\ObjectSpawner\ directory. That file contains a parameter called FileNamePattern that determines the location of the object-spawner log, as shown in this example:

Why

Here are some reasons why you should check the object-spawner log:

1. Performance problems occur within stored processes.
2. Servers do not start or servers stop working. The stored process server, the workspace server, and the pooled workspace server are all started by the object spawner.
3. Running a stored process returns no results and no errors.
4. Users do not have permission to start the server or do not have ReadMetadata permission on their application server context (for example, the SASApp application server).
5. The connecting user has to wait longer than the availability time-out of 60 seconds, and the stored process fails.
6. A server is not available on which to run a stored process.

The following error is an example of one that you might find in this log:

The launch of the server process failed due to a problem with the processing of the SAS logging facility configuration file (LOGCONFIGLOC).

This error might occur because the file that is specified for the -LOGCONFIGLOC option cannot be processed. The option either is invalid or it cannot be accessed.

For more examples of errors that you might see in the object-spawner log, see the section Object Spawner Messages in "Appendix 1: Object Spawner and SAS OLAP Server Messages" in the SAS^® 9.4 Intelligence Platform: Application Server Administration Guide.

You can also enable more detailed logging within the object-spawner log by following the steps in Enable More Detailed Logging for SAS Object Spawner Troubleshooting in "Chapter 10, Administering Logging for SAS Servers" of the SAS^® 9.4 Intelligence Platform: System Administration Guide, Fourth Edition.

Example and possible cause

Let’s look at an example of how the object-spawner log can be helpful.

Suppose that the stored-process server fails to validate. Within the object-spawner log, you might see an error like the following:

Error authenticating user domain\sassrv in function LogonUser. Error 1326 (The user name or password is incorrect. ).
 
The credentials specified for the SASApp – Stored Process Server (A5WN99NR.AZ000007) server definition failed to authenticate. Therefore this server definition will not be included.

With this type of error, you might have an invalid sassrv password. You can check the password by trying to log on to the server directly with the sassrv user ID. If the logon is successful, open SAS^® Management Console and verify that the password is correct in the metadata, as follows:

1. In SAS^® Management Console, select User Manager.
2. Locate the SAS General Servers group. Then, right-click and select Properties.
3. Click the Accounts tab.
4. Select the sassrv user ID. Then click Edit.
5. Update the password to what was successful on the operating system.
6. Restart the object spawner.
7. Check the logs again to verify that the error is resolved.

Object-spawner console log

What

The object-spawner console log, available only on UNIX platforms, does not actually contain information about the object spawner. Instead, this log contains STDERR and STDOUT messages about the applications that are launched by the object spawner. The syntax for this log's name ObjectSpawner_console_machine-name.log.

Where

Under UNIX, the default location for the object-spawner console log is SAS-configuration-directory/Lev1/ObjectSpawner/Logs.

Why

Here are some reasons why you should check this log:

• SAS modules are missing. The program tries to run SAS modules that are not present in your environment.
• Possible memory issues occur.
• Storage or space issues occur.
• Possible encoding issues occur.
• Library permission issues occur.

The following error is an example of one that you might find in the object-spawner console log:

ERROR: Could not find extension: (sasgis)

This error can occur when someone or some component runs code that either checks for the existence of certain SAS modules or that contains references to certain SAS modules that are not present in your environment. Because these modules do not exist, errors are written to the log. This behavior is expected, and it is not indicative of any larger issue. So, you can ignore these messages.

Example and possible cause

Let’s look at an example of how the object-spawner console log can be helpful.

Suppose that your program generates a segmentation violation, but there is no indication as to why that happens. Within the object-spawner console log, you might see an error like the following:

ERROR: No space left on device

This type of error is an indication that you might be running out of disk space. Try adding more space, point to a network drive, or use smaller files.

SAS^® Stored Process Server log

You need to enable verbose logging to see every stored process that runs through the server. For more details about verbose logging, see SAS Note 34114, "Creating a detailed SAS^® Stored Process Server log by default."

What

The syntax for the SAS Stored Process Server log name is SASApp_STPServer_yyyy-mm-dd_machine-name_process-ID.log.

Where

The default locations for the stored-process server log are as follows:

• Windows: SAS-configuration-directory\Lev1\SASApp\StoredProcessServer\Logs
• UNIX: SAS-configuration-directory/Lev1/SASApp/StoredProcessServer/Logs

Why

Here are the reasons that you might need to check this log:

• Performance problems occur.
• You do not receive any results. That is, the stored process runs, but it does not return an error, a warning, or the expected results.

Here is an example of an error that you might see in the log:

ERROR: A lock is not available for library.dataset.data
ERROR: Lock held by process 26834

This error indicates that the data set that is shown in the error cannot be locked for processing because the lock is held by another process. Further investigation of the stored-process server log that contains process ID 26834 shows the DATA step that is also reading from the same data set, which causes the lock.

Here is another error that you might see in this log:

ERROR: No logical assign for filename _WEBOUT.

Possible causes

The two main reasons for this error are as follows:

• The %STPBEGIN and %STPEND macros are enabled.
• The item Package is selected under the Result capabilities section (via Execution Options ► Result capabilities in either SAS Management Console or SAS Enterprise Guide)

The object-spawner console log is often needed in conjunction with the object-spawner log so that you can evaluate the communication between them.

SAS^® Stored Process web-application log

What

The SAS Stored Process web-application log resides on the middle-tier machine, and the syntax for the log's name is SASStoredProcess9.4.log. (Note: The 9.4 in SASStoredProcess9.4.log will change with new releases.)

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\Logs\SASServer1_1
• UNIX: SAS-configuration-directory/Lev1/Web/Logs/SASServer1_1

Why

Here are some scenarios for which you might want to check this log:

• HTTP errors are displayed in the browser when you submit the stored process from the SAS Stored Process Web Application.
• Pages in the stored-process web application (for example, the welcome page, index page, or custom input form) do not load or they take a long time to load.
• A dynamic prompt cannot load.

Examples

The following error occurs when dynamic prompts from a data set and the SAS General Servers user group are denied ReadMetadata permission in SAS Management Console for that specific stored process:

2021-07-30 15:01:43,494 [tomcat-http--10] WARN [sasdemo] com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider - Cannot resolve data type
2021-07-30 15:01:43,505 [tomcat-http--10] ERROR [sasdemo] com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider - Unable to find physical table
com.sas.storage.exception.ServerConnectionException: Unable to find physical table
	at com.sas.prompts.valueprovider.dynamic.DataProviderUtil.getLibrary(DataProviderUtil.java:163)
	at com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider.setConnection(PromptColumnValueProvider.java:815)
	at com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider.getValuesAsList(PromptColumnValueProvider.java:647)
	at com.sas.prompts.valueprovider.dynamic.workspace.PromptColumnValueProvider.getValues(PromptColumnValueProvider.java:633)

The following pop-up message occurs because the SAS General Servers user group or the sasdemo user ID are denied permission to the data set that is needed for the prompts for a stored process.

When you run a stored process from the SAS Stored Process Web Application, it is often helpful to add debugging options to the end of the URL by using the reserved macro variable _DEBUG. The following example URL demonstrates how to use the TRACE and LOG options to obtain pertinent information in the log that is produced in the browser.

http://your.web.server:8080/SASStoredProcess/do?_program=/STP_Examples/test1&_debug=trace,log

For a complete list of _DEBUG= values, see List of Valid Debugging Keywords in "Chapter 7: Building a Web Application with SAS^® Stored Processes" in the SAS^® 9.4 Stored Processes: Developer's Guide, Third Edition.

Workspace-server log

What

You must request logging for the workspace-server log because it is not one that is enabled, by default. The syntax for the log's name is SASApp_WorkspaceServer_yyyy-mm-dd_machine-name_process-ID.log.

Where

As mentioned earlier, you must request logging for this log, as follows:

• Windows: In the sasv9_usermods.cfg file that resides in SAS-configuration-directory\Lev1\SASApp\WorkspaceServer\ directory, add the following command to turn on logging:

  -logconfigloc SAS-configuration-directory\Lev1\SASApp\WorkspaceServer\logconfig.trace.xml"

  This setting generates the log file in SAS-configuration-directory\Lev1\SASApp\WorkspaceServer\Logs.

• UNIX: In the sasv9_usermods.cfg file that resides in SAS-configuration-directory/Lev1/SASApp/WorkspaceServer/logconfig.trace.xml"
  to /Lev1/SASApp/WorkspaceServer/ directory, add the following command to turn on logging:

  -LOGCONFIGLOC SAS-configuration-directory/Lev1/SASApp/WorkspaceServer/logconfig.trace.xml"

  This setting generates the log file that resides in SAS-configuration-directory/Lev1/SASApp/WorkspaceServer/Logs.

Why

If you run the stored process from the SAS Stored Process Web Application with the server set to the workspace server, you need to consult this log for any errors or warnings. Running the stored process from SAS Enterprise Guide produces a log directly in the application. You need to consult the workspace server log in either of these circumstances:

• when Default server is selected as the server type in the stored-process properties
• when you submit the stored process from SAS Enterprise Guide

Pooled workspace-server Log

What

The syntax for the pooled workspace-server log's name is SASApp_PooledWSServer_yyyy-mm-dd_machine-name_process-ID.log.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\ Lev1\SASApp\PooledWorkspaceServer\Logs
• UNIX: SAS-configuration-directory/Lev1/SASApp/PooledWorkspaceServer/Logs

Why

The pooled workspace server loads dynamic prompt values in the SAS Stored Process web application. So, you need this log if a dynamic prompt does not load.

SAS^® Metadata Server Log

What

The syntax for the SAS Metadata Server log' name is SASMeta_MetadataServer_yyyy-mm-dd_machine-name_process-ID.log

Where

The default locations for the metadata-server log are as follows:

• Windows: SAS-configuration-directory\Lev1\SASMeta\MetadataServer\Logs
• UNIX: SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Logs

Why

This log is helpful for issues with the LIBNAME META engine and metadata permissions.

The following error is written to the metadata-server log when no metadata identity is associated with the user name and password.

ERROR [00000779] 28:billyw - User Folders cannot be created or retrieved if connected user ID is not a person identity. Must be connected as a person identity or valid person name must be passed in request.

The following messages might also be written to this log when you try to open SAS Management Console and when the user billyw requires log on as a batch job user rights at the operating system level.

20100122:13.00.36.82: 00002410:NOTE:    User tbanks probably does not have the right to log on as a batch job.
20100122:13.00.36.82: 00002410:ERROR:   Error authenticating user tbanks in function LogonUser.  Error 1385 (Logon failure: the user has not been granted the requested logon type at this computer. ).
20100122:13.00.36.82: 00002410:ERROR:   Access denied.

Infrequently Used Logs

Windows Event Viewer log

The Windows Event Viewer contains several logs that are used less frequently than the aforementioned logs. This fact in no way diminishes their effectiveness when you are troubleshooting stored-process issues. SAS Technical Support will request these logs from you for an issue if it is necessary.

Although the event viewer contains several logs, SAS Technical Support reviews only the application log, the system log, and the security log for errors that are related to start-up and other failures in the SAS Stored Process Server (or any SAS server).

If the sassrv user ID cannot write to the Logs directory, you receive a message in this log.

If a server is stopped or started, those behaviors register in these logs. Hopefully, this review of the various logs that are related to debugging stored-process issues will assist your troubleshooting efforts.

ERROR_yyyy-mm-dd-time.log

What

This web-server log is useful when you have catastrophic failures in loading or running a stored process in a SAS web application. The syntax for this log's name is ERROR_yyyy-mm-dd-time.log.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebServer\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebServer/logs

Why

Here are some reasons why you should check this log:

• Java errors are returned in the browser when you try to load or run a stored process using the SAS Stored Process Web Application.
• Connection errors occur between the web server and the tc Server.

Here is an example of an error that you might find in this log:

[error] (OS 10061)No connection could be made because the target machine actively refused it.  : proxy: HTTP: attempt to connect to 192.168.x.xxx:8080 (machine-name) failed

This error might occur when your servers are down or when they are restarting and are not active yet.

Example and possible cause

Let’s look at an example of how the ERROR_yyyy-mm-dd-time log can be helpful.

Suppose that you try to run a stored process through the SAS Stored Process Web Application and you cannot load any SAS 9.4 web applications after you enter your user ID and password. If you check this log, you might see errors like the following:

No connection could be made because the target machine actively refused it. : proxy: HTTP: attempt to connect to 192.168.x.xxx:8080 (machine-name) failed
 
ap_proxy_connect_backend disabling worker for (machine-name)

The secondary middle-tier node is in a clustered environment, but it configured incorrectly. For a circumvention, see SAS Note 55904, "You cannot access any SAS® 9.4 web applications when the secondary middle-tier node is in a clustered environment."

Localhost_access_log..yyyy-mm-dd.txt

What

This log shows the sequence and query strings of URLs that are submitted through the SAS Web Application Server (for all web applications that use this application server). This log lists activity sequentially and includes HTTP status codes for each URL. The syntax for this log's name is Localhost_access_log..yyyy-mm-dd.txt.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebAppServer\SASServer1_1\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/logs

Why

Here are some reasons why you should check this log:

• A page cannot load or the URL is invalid.
• Performance problems occur.
• You need a good way to trace the stored-process activity after you click the Run button in web applications only.

Here is an example of a message that you might see in this log:

"POST /SASLogon/v1/tickets HTTP/1.1" 404 796

Example and possible cause

Let’s look at an example of how this log can be helpful. Suppose that you cannot log on to the SAS Stored Process Web Application. If you check this log, you might see a message like the following:

GET /SASStoredProcess/j_spring_cas_security_check?ticket=ST-9994-zpD2sFdBmayXEuoj4cya-cas HTTP/1.1" 401 802

In this example, you can see that the issue is a 401 HTTP return code.

If the servers have just been restarted, try waiting a little longer for them to start up. If that is not the issue, the server might be down. The best approach is to stop and then restart the servers. Because of dependencies, it is important to start the servers in the correct order. You can find the correct order in Overview of Server Operation, in "Chapter 6: Operating Your Servers" of the SAS^® Intelligence Platform: System Administration Guide, Fourth Edition.

Access_yyyy-mm-dd-time.log

What

This log shows the sequence and query strings of URLs that are submitted through the SAS Web Server. The syntax for the name of this log is access_yyyy-mm-dd-time.log.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebServer\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebServer/logs

Why

Here are some reasons why you would check this log:

• A connection error occurs when you submit a stored process.
• Java errors occur in the browser.
• When you evaluate a performance problem, timestamps in the log confirm when the web server received the request and what was in the request.
• The output that is produced by the stored process does not match the expected output based on the prompt selections that are available when you submit the stored process.

Here is an example of a message that you might see in this log:

"GET /SASTheme_default/themes/ThemeXMLFiles.config HTTP/1.1" 503 299

This error might occur because the servers are down or are in the process of restarting and are not active yet.

Example and possible cause

Let’s look at an example of how the access_yyyy-mm-dd-time log can be helpful.

Suppose that you are trying to run a stored process through the SAS web application and you receive a connection error. If you check this log, you might see a message like the following:

"GET /SASLogon/proxy?pgt=TGT-6-aGakal9b2VGBg3dnNTbbwiVALOWqBSem9E3cVWehDD35vegmxI-cas&targetService=http%3A%2F%2FLazySecurityContext HTTP/1.1" 502 407

The message above shows a 502 HTTP return code. This error indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server. If the servers were just restarted, try waiting a little longer for them to start back up. If that is not the issue, then a server might be down. The best approach is to stop and restart the servers. Because of dependencies, it is important to start the servers in the correct order. You can find the correct order in Overview of Server Operation, in "Chapter 6: Operating Your Servers" of the SAS^® Intelligence Platform: System Administration Guide, Fourth Edition.

Server.log

What

This log is helpful if a stored-process web application generates an HTPP, browser, or generic error in the browser when it loads a page or results that are returned from a stored process.

Where

The default locations for this log are as follows:

• Windows: SAS-configuration-directory\Lev1\Web\WebAppServer\SASServer1_1\logs
• UNIX: SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/logs

Why

Here are some reasons why you would check this log:

• The SAS Stored Processes web application is not working.
• Requests time out.
• A configuration change is made to one of the web applications (for example, an increase to the time-out for the stored-process web application).
• A generic error is displayed in the SAS Stored Processes web application when you run a stored process.

Here is an example of a message that you might find in this log:

java.io.IOException: Cannot bind to URL

One possible reason for this type of error is network issues at the site where the stored process is run.

Example and possible cause

Let’s look at an example of how the server.log file can be helpful.

Suppose that you want to run a stored process through the SAS Web Application and you receive a generic error that says The system is experiencing problems. Please contact your system administrator. If you check this log, you might see a message like the following:

ERROR (ContainerBackgroundProcessor[StandardEngine[Catalina]]) [org.apache.catalina.core.ContainerBase] Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: Java heap space

There is insufficient memory available to process the request. As a possible workaround, you need to modify the setenv.bat file that resides on Windows in SAS-configuration-directory\Lev1\Web\Webappserver\Sasserver1_1\bin\ or the setenv.sh in SAS-configuration-directory/Lev1/Web/WebAppServer/Sasserver1_1/bin/ on UNIX. You need to edit the JVM_OPTS value by changing the -Xmx4096m -Xms1024m parameter to the following:

-Xmx8092m -Xms8092m

Making this change requires the restart of the SAS Web Application Server if only the webappserver script is updated.

Conclusion

Hopefully, this review of logs that are related to debugging stored-process issues will assist your troubleshooting efforts. When you request help from SAS Technical Support, these are some of the logs that you will be asked to send so that Technical Support can better determine the cause of your problem.

Learn more

• Jobs: Stored processes in Viya
• Creating Stored Processes - SAS Support
• Developer's Guide: SAS Stored Processes

Debugging a stored-process problem was published on SAS Users.

I am pretty sure you have never heard of the TRIMS function, and I would be genuinely surprised if you told me otherwise. This is because this function does not exist (at least at the time of this writing).

But don’t worry, the difference between "nonexistence" and "existence" is only a matter of time, and from now it is less than a blog away. Let me explain. Recently, I published two complementary blog posts:

[1] Removing leading characters from SAS strings

[2] Removing trailing characters from SAS strings

While working on these pieces and researching “prior art” I stumbled upon a multipurpose function in the SAS FedSQL Language that alone does either one or both of these things – remove leading or/and trailing characters from SAS strings.

FedSQL Language and Proc FedSQL

The FedSQL language is the SAS proprietary implementation of the ANSI SQL:1999 core standard. Expectedly, the FedSQL language is implemented in SAS by means of the FedSQL procedure (PROC FEDSQL). This procedure enables you to submit FedSQL language statements from a Base SAS session, and it is supported in both SAS 9.4 and SAS Viya.

Using the FEDSQL procedure, you can submit FedSQL language statements to SAS and third-party data sources that are accessed with SAS and SAS/ACCESS library engines. Or, if you have SAS Cloud Analytic Services (CAS) configured, you can submit FedSQL language statements to the CAS server.

FedSQL TRIM function

FedSQL language has its own vast FedSQL Functions library with hundreds of functions many of which replicate SAS 9.4 Functions. Many, but not all. Deep inside this FedSQL functions library, there is a unique treasure modestly called TRIM Function which is quite different from the BASE SAS Language TRIM() function.

While SAS 9.4 BASE TRIM() function capabilities are quite limited - it removes just trailing blanks from a character string, the FedSQL TRIM() function is way much more powerful. This triple-action function can remove not just trailing blanks, but also leading blanks, as well as both, leading and trailing blanks. On top of it, it can remove not just blanks, but any characters (although one character at a time). See for yourself, this function has the following pretty self-explanatory syntax:

Here trim-character specifies one character (in single quotations marks) to remove from column. If trim-character is not specified, the function removes blanks.

While being called a function, it does not look like a regular SAS function where arguments are separated by commas.  It looks more like an SQL statement (which it understandably is – it is part of the FedSQL language). However, this function is available only in PROC FEDSQL; it’s not available in SAS DATA steps or other PROC steps. Still, it gives us pretty good idea of what such a universal function may look like.

User-defined function TRIMS to remove leading or/and trailing characters in SAS strings

Let’s build such a function by means of the PROC FCMP for the outside the FedSQL usage (it is worth noticing that the FCMP procedure is not supported for FedSQL). To avoid confusion with the existing TRIM function we will call our new function TRIMS (with an ‘S’ at the end) which suits our purpose quite well denoting its plural purpose. First, we define what we are going to create.

The following user-defined function implementation is based on the coding techniques described in the two previous posts, [1] and [2] that I mentioned above. Here goes.

 
libname funclib 'c:\projects\functions';
 
/* delete previous function definition during debugging */
options cmplib=funclib.userfuncs;
proc fcmp outlib=funclib.userfuncs.package1;
   deletefunc trims;
run;
 
/* new function defintion */
proc fcmp outlib=funclib.userfuncs.package1;
   function trims(f $, str $, clist $, mod $) $32767;
      from = 1;
      last = length(str);
      if upcase(f) in ('L', 'B') then from = findc(str, clist, 'K'||mod);
      if from=0 then return('');
      if upcase(f) in ('T', 'B') then last = findc(str, clist, 'K'||mod, -last); 
      if last=0 then return('');
      return(substr(str, from, last-from+1));      
   endfunc; 
run;

Code highlights

• In the FUNCTION statement we assign the length of the returned value as $32767 to accommodate the longest possible character string.
• In the function definition, we first assign initial values of the target substring positions as from=1 and last=length(str).
• Then for Leading or Both character removal, we calculate an adjusted value of from as a position of the first character in str that is not listed in clist and not defined by the mod
• If from=0 then we return blank and stop further calculations as this means that ALL characters are to be removed.
• Then for Trailing or Both character removal, we calculate an adjusted value of last as a position of the last character in str that is not listed in clist and not defined by the mod
• If last=0 then we return blank and stop further calculations as this means that ALL characters are to be removed.
• And finally, we return a substring of str starting at the from position and ending at the last position, that is with the length of last-from+1.

TRIMS function usage

Let’s define SAS data set SOURCE as follows:

data SOURCE;
   input X $ 1-30;
   datalines;
*00It's done*2*1**-
*--*1****9*55
94*Clean record-*00
;

In the following DATA step, we will create three new variables with removed leading (variable XL), trailing (variable XT) and both - leading and trailing (variable XB) characters '*' and '-' as well as any digits:

options cmplib=funclib.userfuncs;
data TARGET;
   set SOURCE;
   length XB XL XT $30;
   XB = trims('b', X, '*-', 'd');
   XL = trims('L', X, '*-', 'd');
   XT = trims('t', X, '*-', 'd');
run;

In this code we use the TRIMS function three times, each time with a different first argument to illustrate how this affects the outcome.

Usage highlights

• The first argument of the TRIMS function specifies whether we remove characters from both leading and trailing positions ('b'), from leading positions only ('L'), or from trailing positions only ('t'). This argument is case-insensitive. (I prefer using capital 'L' for clarity since lowercase 'l' looks like digit '1').
• The second argument specifies the name of the variable (X) that we are going to remove characters from (variable X is coming from the dataset SOURCE).
• The third argument '*-' specifies which character (or characters) to remove. In our example we are removing '*' and '-'. If you do not need to explicitly specify any character here, you still must supply a null value ('') since it is a required argument. In this case, the fourth argument (trim-list-modifier) will determine the set of characters to be removed.
• The fourth argument (case-insensitive) of the TRIMS function specifies the FINDC function modifier(s) to remove certain characters in bulk (in our example 'd' will remove all digits). If such modifier is not needed, you still must supply a null value ('') since all four arguments of the TRIMS function are positional and required.
• Please note that before calling the TRIMS function we assign the lengths for the variables XB, XL, and XT. Had we not done that, these variables would be assigned the maximum length of 32,767 bytes as specified in the TRIMS function definition.

Here is the output data table TARGET showing the original string X and the resulting strings XB (Both leading and trailing characters removed), XL (Leading characters removed) and XT (Trailing characters removed) side by side:

Conclusion

The new TRIMS function presented in this blog post goes ways further the ubiquitous LEFT, TRIM, and STRIP functions that remove the leading (LEFT), trailing (TRIM), or both - leading and trailing (STRIP) blanks. The TRIMS function handles ANY characters, not just blanks. It also expands the character deletion functionality of the powerful  FedSQL TRIM function beyond just removing any single leading and/or trailing character. The TRIMS function single-handedly removes any number of explicitly specified characters from either leading, trailing or both (leading and trailing) positions. Plus, it removes in bulk many implicitly specified characters. For example 'd' modifier removes all digits, 'du' modifier removes all digits ('d') and all uppercase letters ('u'), 'dup' modifier removes all digits ('d'), all uppercase letters ('u') and all punctuation marks ('p'); and so on as described by the FINDC function modifiers. The order in which modifier characters are listed does not matter.

Additional resources

• Removing leading characters from SAS strings
• Removing trailing characters from SAS strings
• Deleting a substring from a SAS string
• Removing repeated characters in SAS strings

Questions? Thoughts? Comments?

Do you find this post useful? Please share your thoughts with us below.

Introducing TRIMS function to remove any leading and/or trailing characters from SAS strings was published on SAS Users.

All computer data is stored as a series of 1s and 0s. For example, an uppercase A in ASCII is 01000001. Many encrypting schemes use a key to transform plaintext into ciphertext. As an example, suppose your key is B (01000010).

I'm sure you remember the Boolean operators AND, NOT, and OR. If you perform an AND operator on A and B, the result is a 1 if both values are true (1) and false (0) otherwise. So, A AND B is 01000000. The OR operator results in a value of true if either A or B is true, or both A and B are true. Therefore, A OR B is 01000011. The operator that you might not be as familiar with is the exclusive OR (XOR) operator. This is similar to an OR operator except that if both A and B are true, the result is false. A XOR B is equal to 00000011. Why is this useful? An interesting property of the XOR operator is if you take the result and use the XOR operator again on the previous result, you get back to the original value.

The table below shows how the XOR operator works:

If A is the cleartext message and B is the key, A XOR B is the ciphertext. If you perform an exclusive OR with the key (B) and the ciphertext (as was done in the last line), you get back the cleartext value (A).

Of course, you don't use single letter keys to encode messages. Key lengths of 8, 16, or even up to 512 are common. The problem with any key is that if you apply it to a long message, there is a pattern in the ciphertext that allows a code breaker to figure out how long the key is, and even what it is. There are several computer programs that can decode many of the popular encrypting methods. My favorite is:

https://www.geocachingtoolbox.com/

Here is a list of ciphers that can be broken with this program:

Let's write a short SAS program that uses a key to encode a text string.

data Encode;
 
retain Key 12345; ❶
 
length Letter $ 1; ❷
 
String = 'This is a test';
 
do i = 1 to lengthn(String);
 
Letter = substr(String,i,1); ❸
 
Rank = rank(Letter); ❹
 
Coded = bxor(Rank,Key); ❺
 
Decoded = bxor(Coded,Key); ❻
 
Clear = byte(Decoded); ❼
 
output;
 
end;
 
drop i;
 
run;
 
 
 
title "Listing of Data Set Encode";
 
proc print dta=Encode noobs;
 
var Letter Key Rank Coded Decoded Clear;
 
format Key Rank Coded binary8.;
 
run;

❶ A RETAIN statement is used to assign the number 12345 to a numeric variable called Key.  (You could have used an assignment statement, but using a RETAIN statement is more efficient and elegant.

❷ The variable Letter will hold each letter of the message and is set to a length of one.

❸ The SUBSTR function will extract each letter from String.

❹ Because Boolean operators only operate on true/false values, you use the RANK function to convert each letter to its ASCII value (stored internally as a series of 0s and 1s).

❺ You now use the BXOR (binary exclusive OR) function to encode each letter of your message.

❻ To demonstrate that the program is working, you use the BXOR function again to demonstrate that this process will return the original String.

❼ The BYTE function takes an ASCII value and returns the appropriate character.

Here is the listing of data set Encode:

Because this encryption method uses a single (and short) key, it would be fairly easy to break. What if you encode every letter of the original message with a different key? You can accomplish this by using the SAS random function RAND and using a seed value so that the same series of random numbers gets generated every time you run the program. You can even use one of a dozen different random distributions, to make it harder for someone to decode your file. Here is an example:

First, here is a copy of my text file that contains my secret message (stored in the location c:\Books\Blogs\Cipher\Clear_Text.txt).

Good morning Mr. Phelps.

Your mission, should you decide to accept it,

is to rid the world of evil.

As usual, if you or any member of your team are caught or killed,

the Secretary will disavow any knowledge of your actions.

The following program encrypts this file and creates a temporary data set (in a real situation, you would make this a permanent data set):

data Coded;
 
call streaminit(13579); ❶
 
array l[150] $ 1 _temporary_; ❷
 
array num[150] _temporary_; ❸
 
array xor[150]; ❹
 
infile "c:\Books\Blogs\Cipher\Clear_Text.txt" pad;
 
input string $150.; ❺
 
do i = 1 to dim(l);
 
l[i] = substr(string,i,1); ❼
 
num[i] = rank(l[i]); ❽
 
xor[i] = bxor(num[i],int(100*rand('Uniform'))); ❾
 
end;
 
keep xor1-xor150;
 
run;

❶ You need to set a seed value using CALL STREAMINIT so that when you run the decoding program, you will generate the same series of random numbers.

❷ This temporary array will hold up to 150 characters.

❸ The Num array holds the numerical ASCII value for each of the characters in the line.

❹ The XOR array holds the values of the exclusive OR between each numerical ASCII value and the Key.

❺ Read in a string of up to 150 characters.

❻ The DO LOOP picks up each character in a line, starting from 1 and ending at the length of each line.

❼ The RANK function outputs the ASCII value of each character.

❽ The BXOR (binary exclusive OR) function performs the exclusive OR between each ASCII value and the Key.

To decode this message, use the following program:

data Decode;
 
call streaminit(13579); ❶
 
array l[150] $ 1 _temporary_;
 
array num[150] _temporary_;
 
array xor[150];
 
length String $ 150;
 
set Coded;
 
do i = 1 to dim(l);
 
num[i] = bxor(xor[i],int(100*rand('Uniform'))); ❷
 
l[i] = byte(num[i]); ❸
 
substr(String,i,1) = l[i]; ❹
 
end;
 
keep String;
 
run;

❶ Notice that the value of the CALL STREAMINIT routine uses the same seed as the previous program.

❷ The BXOR function between each coded value and the Key, will produce the cleartext.

❸ The BYTE function will convert the ASCII values back to letters, numbers, and other characters.

❹ Finally, the SUBSTR function used on the left-hand side of the equal sign will place each of the characters into the appropriate location in the String variable. (See my previous blog that discusses the use of the SUBSTR function used on the left-hand side of the equal sign.)

Here is the output:

It would be straightforward to convert these two programs into macros so that you could encrypt and decrypt any file.

Of course, I would be remiss if I didn't mention that you can encrypt a SAS data set using two data set options ENCRYPT=and ENCRYPTKEY="password". But what would be the fun of that?

Here is an example:

*Note: AES stands for Advanced Encryption Standard
 
If you use quotation marks on the ENCRYPTKEY= option, you have
 
more flexibility in choosing a password (maximum length=64);
 
 
 
data Secret(encrypt=aes encryptkey="mypassword");
 
input String $80.;
 
datalines;
 
This is a secret message.
 
See if you can decode it.
 
This message will not self-destruct!
 
;

You can decode the encrypted data set by including the DATA set option ENCRYPTKEY="password" in any procedure, such as the PROC PRINT shown below:

title "Listing of Data Set Secret";
 
proc print data=Secret(encryptkey="mypassword") noobs;
 
run;

Encrypting Data Using SAS was published on SAS Users.

As artificial intelligence comes of age and data continues to disrupt traditional industry boundaries, the need for real-time analytics is escalating as organizations fight to keep their competitive edge. The benefits of real-time analytics are significant. Manufacturers must inspect thousands of products per minute for defects. Utilities need to eliminate unplanned downtime to keep the lights on and protect workers. And governments need to warn citizens of natural disasters, like flooding events, providing real time updates to save lives and protect property.

Each of these use cases requires a complex network of IoT sensors, edge computing, and machine learning models that can adapt and improve by ingesting and analyzing a diverse set of high-volume, high-velocity data.

SAS and Microsoft are partnering to inspire greater trust and confidence in every decision, by innovating with proven AI and streaming analytics in the cloud and on the edge. Together, we make it easier for companies to harness hidden insights in their diverse, high volume, high velocity IoT data, and capitalize on those insights in Microsoft Azure for secure, fast, and reliable decision making.

To take advantage of all the benefits that real-time streaming analytics has to offer, it’s important to tailor your streaming environment to your organization’s specific needs. Below, we’ll dive into how to understand the value of IoT in parallel to your organization’s business objectives and then strategize, plan, and manage your streaming analytics environment with SAS Viya on Azure.

Step 1: Understand the value of IoT

While you may already know that IoT and streaming analytics are the right technologies to enable your business’ real time analytics strategy, it is important to understand how it works and how you can benefit. You can think of streaming analytics for IoT in three distinct parts: sense, understand and act.

• Sense: Sensors by design are distributed, numerous, and collect data at high fidelity in various formats. The majority of data collected by sensors has a short useful life and requires immediate action. Streaming analytics is well-suited to this distributed sensor environment to collect data for analysis.
• Understand: A significant number of IoT use cases requires quick decision-making in real time or near-real time. To achieve this, we need to apply analytics to data in motion. This can be done by deploying AI models that detect anomalies and patterns as events occur.
• Act: As with any analytics-based decision support, it is critical to act on the insight generated. Once a pattern is detected this must trigger an action to reach a desired outcome. This could be to alert key individuals or change the state of a device, possibly eliminating the need for any human intervention.

The value in IoT is driven by the reduced latency to trigger the desired outcome. Maybe that’s improving production quality in the manufacturing process, recommending a new product to a customer as they shop online, or eliminating equipment failures in a utility plant. Whatever it is, time is of the essence and IoT can help get you there.

Step 2: Strategize

Keeping the “sense, understand, act” framework in mind, the next step is to outline what you hope to achieve. To get the most out of your streaming analytics with SAS and Microsoft, keep your objectives in mind so you can stay focused on the business outcome instead of trying to act on every possible data point.

Some important questions to ask yourself are:

1. What is the primary and secondary outcomes you are hoping to achieve? Increase productivity? Augment safety? Improve customer satisfaction?
2. What patterns or events of interest do you want to observe?
3. If your machines and sensors show anomalous behavior what actions need to be taken? Is there an existing business process that reflects this?
4. What data is important to be stored as historical data and what data can expire?
5. What kind of infrastructure exists from the point where data is generated (edge) to cloud? Is edge processing an option for time-critical use cases or does processing needs to be centralized in cloud?
6. What are your analytics and application development platforms? Do you have access to high performance streaming analytics and cloud infrastructure to support this strategy?

Once you’ve identified your outcomes, define which metrics and KPIs you can measure to show impact. Make sure to have some baseline metrics to start from that you can improve upon.

Step 3: Plan and adopt

Now it’s time to take your strategy and plan the adoption of streaming analytics across your business.

Adoption will look different if you already have an IoT platform in place or if you are working to create a net-new solution. If you are going to be updating or iterating upon an existing solution, you will want to make sure you have access to key historical data to measure improvement and use institutional knowledge to maximize performance. If you are working with a net-new solution, you will want to give yourself some additional time to start small and then scale your operations up over time so you can tackle any unforeseen challenges.

In both cases it is important to have key processes aligned to the following considerations:

• Data variety, volume and accuracy: Focus here on the “sense” part of the “sense, understand, act” framework. Accessing good data is the foundation to the success of your streaming projects. Make sure you have the right data needed to achieve your desired business outcome. Streaming analytics helps you understand the signals in IoT data, so you can make better decisions. But if you can’t access the right data, or your data is not clean, your project will not be successful. Know how much data you will be processing and where. Data can be noisy, so it is important to understand which data will give you the most insight.
• Reliability: Ensure events are only processed once so you’re not observing the same events multiple times. When equipment fails or defects occur on the production line, ensure there are processes in place to auto-start to maximize uptime for operations.
• Scalability: Data science resources are scarce, so choose a low-code, no-code solution that can address your need to scale. When volume increases, how are you going to scale up and out? Azure simplifies scale with its PaaS offerings, including the ability to auto-scale SAS Viya on Azure.
• Operations: Understand how you plan to deploy your streaming analytics models, govern them and decide which processes can be automated to save time.
• Choose the right partners and tools: This is critical to the success of any initiative. SAS and Microsoft provide a best-in-class solution for bringing streaming analytics on the most advanced platform for integrated cloud and edge analytics.

Now that you have created your plan, it is time to adopt. Remember to start small and add layers of capability over time.

Step 4: Manage

To get the most value from IoT and streaming analytics, organizations must implement processes for continuous iteration, development, and improvement. That means having the flexibility to choose the most powerful models for your needs – using SAS, Azure cloud services, or open source. It also means simplifying DevOps processes for deploying and monitoring your streaming analytics to maximize uptime for your business systems.

With SAS Viya on Azure, it is easy to do this and more. Seamlessly move between your SAS and Microsoft environment with single sign on authentication. Develop models with a host of no-code, low-code tools, and monitor the performance of your SAS and open-source models from a single model management library.

Maximizing value from your IoT and streaming analytics systems is a continuous, agile process. That is why it is critical to choose the most performant platform for your infrastructure and analytics needs. Together, SAS and Microsoft make it easier for organizations of all sizes and maturity to rapidly build, deploy, and scale IoT and streaming analytics, maximizing up time to better serve customers, employees, and citizens.

If you want to learn more about SAS and streaming analytics and IoT capabilities as well as our partnership with Microsoft, check out the resources below:

• Learn about SAS Viya’s IoT and streaming analytics capabilities
• Discover out all the exciting things SAS and Microsoft are working to achieve together at SAS.com/Microsoft
• See how SAS and Microsoft work together to help the town of Cary, North Carolina warn citizens of flood events: Smart city uses analytics and IoT to predict and manage flood events

Your guide for analyzing real time data with streaming analytics from SAS® Viya® on Azure was published on SAS Users.

Welcome back to my SAS Users blog series CAS Action! - a series on fundamentals. I've broken the series into logical, consumable parts. If you'd like to start by learning a little more about what CAS Actions are, please see CAS Actions and Action Sets - a brief intro. Or if you'd like to see other topics in the series, see the overview page.

In this example, I will use the CAS procedure to execute the dataStep.runCode CAS action. Be aware, instead of using the CAS procedure, I could execute the action with Python, R, or even a REST API with some slight changes to the syntax for the specific language.

Why use the DATA Step?

It's pretty simple, the DATA step is a powerful way to process your data. It gives you full control of each row and column, ability to easily create multiple output tables, and provides a variety of statements to pretty much do anything you need.

In this example, I will use the DATA step to quickly create three CAS tables based on the value of a column.  Before we execute the DATA step, let's view the frequency values of the Origin column in the cars table. To do that, I'll use the simple.freq action.

proc cas;
    simple.freq / 
          table={name='cars', caslib='casuser'},
           input='Origin';
quit;

The result of the freq action shows that the Origin column in the cars CAS table has three distinct values: Asia, Europe and USA. I can use that information to create three CAS tables based off these unique values using the SAS DATA step.

Execute DATA Step in SAS Viya's CAS Server

One way to execute the DATA step directly in CAS is to use the runCode action with the code parameter. In the code parameter just specify the DATA step as a string. That's it!

In this example, I'll add the DATA step within a SOURCE block. The SOURCE block stores the code as variable. The DATA step code is stored in the variable originTables. This DATA step will create three CAS tables, one table for each unique value of the Origin column in the cars table.

proc cas;
    source originTables;
        data casuser.Asia
             casuser.Europe
             casuser.USA;
            set casuser.cars;
            if Origin='Asia' then output casuser.Asia;
            else if Origin='Europe' then output casuser.Europe;
            else if Origin='USA' then output casuser.USA;
        run;
    endsource;
 
    dataStep.runCode / code=originTables;
quit;

The runCode action executes the DATA step in the distributed CAS environment and returns information about the input and output tables. Notice three CAS tables were created: Asia, Europe and USA.

DATA Step in CAS has Limitations

Now, one thing to be aware of is not all functionality of the DATA step is available in CAS. If you are using the runCode action with an unsupported statement or function in CAS, you will receive an error. Let's look at an example using the first function, which gets the first letter of a string, and is not supported in CAS.

proc cas;
    source originTables;
        data casuser.bad;
            set casuser.cars;
            NewCol=first(Model);
        run;
    endsource;
    dataStep.runCode / code=originTables;
quit;

The results of the runCode action return an error. The error occurs because the FIRST function is unknown or cannot be accessed. In situations like this you will need to find a CAS supported method to complete the task. (HINT: Here instead of the first function you can use the substr function).

For more information visit Restrictions and Supported Language Elements. Be sure to find the version of your SAS Viya environment.

Summary

In SAS Viya, the runCode action provides an easy way to execute most of the traditional DATA step in CAS in any language, from the CAS Language (CASL), to Python, R, Lua, Java and more.

Additional Resources

runCode Action
DATA Step Action Set: Details
Restrictions and Supported Language Elements
SOURCE statement
SAS® Cloud Analytic Services: Fundamentals
Code

CAS-Action! Executing the SAS DATA Step in SAS Viya was published on SAS Users.

I'll demonstrate several different techniques that produce identical results, and compare processing speeds. For a more robust assessment, I'll test the techniques while reading from both SAS data sets and database tables. Analysis of the results clearly shows that when the techniques produce identical results, these choices usually produce faster run times:

1. Use a WHERE statement instead of a subsetting IF statement.
2. Use the KEEP= dataset option on input data sets instead of a KEEP statement.
3. Should I use SQL or DATA step?
   • If producing a single result, use SQL.
   • If producing multiple results, use the DATA step.
4. If your process is CPU bound:
   • If you have access to CAS, run it in CAS.
   • Otherwise, refactor in DS2.

The programs I used to create the video are available for download from GitHub if you want to experiment.

Until next time, may the SAS be with you!
Mark

Jedi SAS Tricks: 5 Ways to Make Your SAS Code Run Faster was published on SAS Users.

Welcome back to my SAS Users blog series CAS Action! - a series on fundamentals. I've broken the series into logical, consumable parts. If you'd like to start by learning a little more about what CAS Actions are, please see CAS Actions and Action Sets - a brief intro. Or if you'd like to see other topics in the series, see the overview page.

In this example, I will use the CAS procedure to execute the execDirect action. Be aware, instead of using the CAS procedure, I could execute the same action with Python, R, or even a REST API with some slight changes to the syntax for the specific language.

Why not PROC SQL?

The FedSQL language is the SAS implementation of the ANSI SQL:1999 core standard. FedSQL provides a scalable, threaded, high-performance way to query data and create tables.  FedSQL also enables you to process and join data using industry-standard query expressions and SQL expressions. In SAS Viya, the execDirect CAS action submits a SAS FedSQL language statement for immediate execution in the CAS server. Moreover, you can also use PROC FEDSQL to query tables in CAS. Behind the scenes the CAS API converts the procedure to the execDirect action.

For my PROC SQL fans, be careful. If you use PROC SQL to query a CAS table, the entire CAS table is sent back to the client for processing. This can cause slow performance or cause a data transfer error because PROC SQL does not execute in CAS.

Execute SQL in SAS Viya

Consider the code below. To execute fedSQL directly in CAS, use the fedSQL.execDirect CAS action with the query parameter. In the query parameter specify the query as a string. In the FROM clause use a two level naming convention to specify the CAS table name and caslib. Here I'll specify the cars table in the Casuser caslib.

proc cas;
    fedSQL.execDirect / 
        query="select Make,
                      Model,
                      MSRP,
                      mean(MPG_City,MPG_Highway) as MPG_Avg
               from casuser.cars
               where Make='Toyota'
               order by MPG_Avg desc";
quit;

And the results:

The execDirect action executes the query in the distributed CAS environment and returns the expected results.

Using a SOURCE block

While the query we executed was simple, this is not always the case. Adding a complicated query as a string can make writing the query difficult.

Instead of specifying the query as a string in the CAS procedure, use the SOURCE statement to embed text in a variable. In the following example, I'll execute the same query as before. However, this time I'll nest the query inside a SOURCE block by specifying the SOURCE statement and name the variable MPG_toyota. Then I'll add the query inside the SOURCE block and use the ENDSOURCE statement to end the block.

proc cas;
    source MPG_toyota;
        select Make,
               Model,
               MSRP,
               mean(MPG_City,MPG_Highway) as MPG_Avg
        from casuser.cars
        where Make='Toyota'
        order by MPG_Avg desc;
    endsource;
 
    fedSQL.execDirect / query=MPG_toyota;
quit;

After the SOURCE block is complete, you can reference the variable as the value to the query parameter in the execDirect action.

And the results:

The results returned are the same, but using a SOURCE block makes the code easier to write and maintain.

Summary

In SAS Viya, FedSQL provides a scalable, threaded, high-performance way to query data and create new CAS tables from existing tables in the CAS server. In this example we saw two distinct ways to run SQL code on SAS Viya. This is only the beginning. See the resources below for more details on PROC FedSQL.

Resources

SAS® Viya®: FedSQL Programming for SAS® Cloud Analytic Services
SAS® Viya®: How to Emulate PROC SQL Using CAS-Enabled PROC FedSQL
FEDSQL Procedure
execDirect CAS Action
SOURCE statement
SAS® Cloud Analytic Services: Fundamentals
Code

CAS-Action! Executing SQL in SAS Viya was published on SAS Users.

Need to authenticate on REST API calls

While researching authentication I ran into a couple of resources on SAS and OAuth. First, what I consider the gold standard in Mike Roda’s SAS Global Forum paper OpenID Connect Opens the Door to SAS Viya APIs. We have also re-written the SAS Viya REST API ‘Getting Started’ documentation on developer.sas.com, which covers details on application registration and access token generation. The rewrite includes sample code for both curl and Python.

Consider this post a quick guide to summarize these resources and shed light on authenticating via authorization code and passwords.

What OAuth grant type should I use?

Choosing the grant method to get an access token with OAuth depends entirely on your application. You can start with this guide Which OAuth 2.0 grant should I implement? to understand the basics of OAuth grant type options. For a SAS perspective, refer to Tara St. Clair’s article Building custom apps on top of SAS Viya, Part Three: Choosing an OAuth flow.

This post covers three grant methods: authorization code, client credentials, and password. Authorization code grants are generally used with web applications and considered the safest choice. The client credentials grant is not associated with any user and is scoped according to the authorities registered to the client. Password grants are most often used by mobile apps and applied in trusted environments. The password grant type is not considered secure and should not be used in production environments.

The process, briefly

Getting an external application connected to the SAS Viya platform requires a series of steps. The process below pertain to the authorization code grant type; however, the steps are similar for all grant types.

1. Use the SAS Viya configuration server's Consul token to obtain an Access Token to register a new Client ID
2. Use the Access Token to register the new client ID and secret
3. Obtain the authorization code
4. Acquire the OAuth access token of the Client ID using the authorization code
5. Call the SAS Viya API using the access token for the authentication.

The process requires multiple roles. A SAS administrator performs steps 1 and 2. A script or app provides the URL to an end user for step 3. This produces an authorization code and the user plugs it back into the script or application, so it can continue.

Registering the client and getting the authorization code (steps 1,2, and 3) are one-time processes. The access and refresh tokens (step 4) are created once and only need to be refreshed if/when the token expires. Once you have the access token, you can call any API (step 5) if your access token is valid.

The process, in detail

Let's look at each grant type in more detail.

Get an access token using an authorization code

For all three grant types covered, you must register the client. The process explained here is in the context of the the authorization code grant type. The client registration process is the same across all grant types, except for the authorization_code parameter. It must match the grant type procedure followed by the user later on.

Step 1: Get the SAS Viya Consul token to register a new client

The first step to register the client is to get the consul token from the SAS server. As a SAS administrator (sudo user), access the consul token using the following commands:

For SAS Viya 3.5 and before

On the SAS Viya server, run the following command to get the Consul token and add it to the CONSUL_TOKEN environment variable:

$ export CONSUL_TOKEN=`cat /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/consul/default/client.token`

And the response: 64e01b03-7dab-41be-a104-2231f99d7dd8 (when not stored in a variable).

For SAS Viya 2020.1 and later

On the SAS Viya cluster, run the following command to get the Consul token (replace value to fit your environment):

kubectl -n <namespace> get secret sas-consul-client -o jsonpath="{.data.CONSUL_TOKEN}" | echo "$(base64 -d)"

And the response: 64e01b03-7dab-41be-a104-2231f99d7dd8

The Consul token returns and is used to obtain an access token used to register the new application. Use the following cURL command to get the token:

$ curl -k -X POST "https://sasserver.demo.sas.com/SASLogon/oauth/clients/consul?callback=false&serviceId=myclientid" \
     -H "X-Consul-Token: $CONSUL_TOKEN"

And the response:

 {
   "access_token":"eyJhbGciOiJSUzI1NiIsIm...",
   "token_type":"bearer",
   "expires_in":35999,
   "scope":"uaa.admin",
   "jti":"de81c7f3cca645ac807f18dc0d186331"
}

Where serviceid in the URL represents the name of the client to register ('myclientid' in this case). The returned token can be lengthy. To assist in later use, create an environment variable from the returned token:

$ export IDTOKEN="eyJhbGciOiJSUzI1NiIsIm..."

Step 2: Register the new client

Change the client_id, client_secret, and scopes in the code below. For detailed guidance on creating and managing Client IDs and Secrets, refer to this page. Scopes should always include "openid" along with any other groups this client needs to get in the access tokens. You can specify "*" but then the user gets prompted for all available groups. The example below just uses one additional group named "group1". Check with your SAS administrator for which scopes to include. You can find more information on scopes and authorities (related parameter for client credentials grant) in this SAS Communities thread.

$ curl -k -X POST "https://sasserver.demo.sas.com/SASLogon/oauth/clients" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $IDTOKEN" \
     -d '{
      "client_id": "myclientid", 
      "client_secret": "myclientsecret",
      "scope": ["openid", "group1"],
      "authorized_grant_types": ["authorization_code","refresh_token"],
      "redirect_uri": "urn:ietf:wg:oauth:2.0:oob"
     }'

And the response:

{
   "scope":[
      "openid",
      "group1"],
   "client_id":"myclientid",
   "resource_ids":[
      "none"],
   "authorized_grant_types":[
      "refresh_token",
      "authorization_code"],
   "redirect_uri":[
      "urn:ietf:wg:oauth:2.0:oob"],
   "autoapprove":[ ],
   "authorities":[
      "uaa.none"],
   "lastModified":1547138692523,
   "required_user_groups":[]
}

Step 3: Approve access to get authentication code

Place the following URL in a browser. Change the hostname and myclientid in the URL as needed.

https://sasserver.demo.sas.com/SASLogon/oauth/authorize?client_id=myclientid&response_type=code

The browser redirects to the SAS login screen. Log in with your SAS user credentials.

SAS Login Screen

On the Authorize Access screen, select the openid checkbox (and any other required groups) and click the Authorize Access button.

Authorize Access form

After submitting the form, you'll see an authorization code. For example, "lB1sxkaCfg". You will use this code in the next step.

Authorization Code displays

Step 4: Get an access token using the authorization code

Now we have the authorization code and we'll use it in the following cURL command to get the access token to SAS.

$ curl -k https://sasserver.demo.sas.com/SASLogon/oauth/token \
     -H "Accept: application/json" -H "Content-Type: application/x-www-form-urlencoded" \
     -u "myclientid:myclientsecret" -d "grant_type=authorization_code&code=YZuKQUg10Z"

And the response:

{
   "access_token":"eyJhbGciOiJSUzI1NiIsImtpZ...",
   "token_type":"bearer",
   "refresh_token":"eyJhbGciOiJSUzI1NiIsImtpZC...",
   "expires_in":35999,
   "scope":"openid",
   "jti":"b35f26197fa849b6a1856eea1c722933"
}

We use the returned token to authenticate and authorize the calls made between the client and SAS. We also get a refresh token we use to issue a new token when the current one expires. This way we can avoid repeating all the previous steps. I explain the refresh process further down.

We will again create environment variables for the tokens.

$ export ACCESS_TOKEN="eyJhbGciOiJSUzI1NiIsImtpZCI6ImxlZ..."
$ export REFRESH_TOKEN="eyJhbGciOiJSUzI1NiIsImtpZC..."

Step 5: Use the access token to call SAS Viya APIs

The prep work is complete. We can now send requests to SAS Viya and get some work done. Below is an example REST call that returns the top level folders.

$ curl -k https://sasserver.demo.sas.com/folders/folders?filter=isNull(parent) \
-H "Authorization: Bearer $ACCESS_TOKEN"

And the (partial) response:

{
    "links": [
        {
            "method": "GET",
            "rel": "collection",
            "href": "/folders/folders",
            "uri": "/folders/folders",
            "type": "application/vnd.sas.collection"
        },
        {
            "method": "GET",
            "rel": "self",
            "href": "/folders/folders?start=0&limit=20",
            "uri": "/folders/folders?start=0&limit=20",
            "type": "application/vnd.sas.collection"
        },
        {
            "method": "POST",
            "rel": "createFolder",
            "href": "/folders/folders",
...

Use the refresh token to get a new access token

To use the refresh token to get a new access token, simply send a cURL command like the following:

$ curl -k https://sasserver.demo.sas.com/SASLogon/oauth/token -H "Accept: application/json" \
     -H "Content-Type: application/x-www-form-urlencoded" \
     -u "myclientid:myclientsecret" \
     -d "grant_type=refresh_token&refresh_token=$REFRESH_TOKEN"

And the response:

{
   "access_token":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImxlZ...",
   "token_type":"bearer",
   "refresh_token":"eyJhbGciOiJSUzI1NiIsImtpZCSjYxrrNRCF7h0oLhd0Y",
   "expires_in":35999,
   "scope":"openid",
   "jti":"a5c4456b5beb4493918c389cd5186f02"
}

Note the access token is new, and the refresh token remains static. Use the new refresh token for future REST calls. Make sure to replace the ACCESS_TOKEN variable with the new token. Also, the access token has a default life of ten hours before it expires. Most applications deal with expiring and refreshing tokens programmatically. If you wish to change the default expiry of an access token in SAS, make a configuration change in the JWT properties in SAS.

Get an access token using the client credentials grant

The steps to obtain an access token using the client credentials grant are similar to the authorization code. I highlight the differences below, without repeating all the steps. The process for registering the client is the same as described earlier, except for the grant type parameter where you will now use "authorized_grant_types": ["client_credentials"]. With client credentials, there is no concept of a refresh token, so it's been removed from the parameter.

Now that the client is registered, the only thing left is to get the access token. Use the code below to get the token:

curl -k -X POST "https://<sas-server>/SASLogon/oauth/token" \
       -H "Content-Type: application/x-www-form-urlencoded" \
       -d "grant_type=client_credentials" \
       -u "myclientid:myclientsecret"

That's it. You can now use the access token retuned in the preceeding call to make REST API calls.

Get an access token using the password grant

The steps to obtain an access token using the password grant are again similar to those describe earlier. When registering the client, use the password grant "authorized_grant_types": ["password","refresh_token"].

The client is now registered on the SAS Viya server. To get the access token, we send a command like we did when using the authorization code and client credentials, just using the username and password.

curl -k https://sasserver.demo.sas.com/SASLogon/oauth/token \
     -H "Content-Type: application/x-www-form-urlencoded" 
     -u "myclientid:myclientsecret" 
     -d "grant_type=password&username=sasdemo&password=mypassword"

And the response:

{
   "access_token":"eyJhbGciOiJSUzI1NiIsImtpZCI6Imx...",
   "token_type":"bearer",
   "refresh_token":"eyJhbGciOiJSUzI1NiIsImtpZ...",
   "expires_in":43199,
   "scope":"DataBuilders ApplicationAdministrators SASScoreUsers clients.read clients.secret uaa.resource openid PlanningAdministrators uaa.admin clients.admin EsriUsers scim.read SASAdministrators PlanningUsers clients.write scim.write",
   "jti":"073bdcbc6dc94384bcf9b47dc8b7e479"
}

From here, sending requests and refreshing the token steps are identical to the method explained in the other code examples.

Final thoughts

At first, OAuth seems a little intimidating; however, after registering the client and creating the access and refresh tokens, the application will handle all authentication components . This process runs smoothly if you plan and make decisions up front. I hope this guide clears up any question you may have on securing your application with SAS. Please leave questions or comments below.

Authentication to SAS Viya: a couple of approaches was published on SAS Users.