By: Jeremy Kadlec | Last Updated: 2017-01-10 | Comments (1) | SQL Server Agent
Our organization spends significant time and money on integrating applications (ERP, CRM, cloud, mainframe, custom, etc.) across our enterprise. We have a number of experts on each platform, but when we need to integrate the data between systems we run into countless issues. How can we leverage our internal application expertise, but ease the integration portion? What options are available from an integration perspective?
There are a number of options to consider when performing enterprise application integration. The first option many companies consider is building a custom application in .NET, Java, etc. to meet a variety of application needs. This provides a tremendous amount of flexibility for the current landscape, but generally requires an organization's top resources to become experts on a number of systems for an extended period of time. That time can extend much longer than expected as the landscape changes with updates needed to support the ever changing integration needs, application upgrades and new platforms being introduced into the enterprise. No business process is static and this technical team would need to work closely with numerous departments in order to be successful. Unfortunately, the time to implement and maintain this custom solution can be far more than expected. That is time when those expert Developers could be innovating new solutions to help the organization be competitive in the market.
A second option many Microsoft based organizations consider is SQL Server Integration Services. Integration Services is designed to extract, transform and load data to / from a variety of systems. In many respects this technology can help SQL Server DBAs and Developers with the data integration between Microsoft platforms, but has some limitations with high speed data integration with non-Microsoft platforms. Often times direct data access is not possible, so workarounds can include moving clear text files around the organization, enabling xp_cmdshell, configuring Linked Servers, etc. Although these are viable options, they could open up security issues, the troubleshooting can be cryptic across numerous platforms, the performance may be less than optimal on certain platforms and code can become unwieldy very quickly.
With these two options the organization and technology group face numerous challenges:
- Time intensive development
- Expert Developers needed
- Proprietary logic
- Multi-platform knowledge
- Insecure data
- Careless security
- Time slicing processes creates inefficiencies
- Tedious troubleshooting
- Scheduling interdependencies
- Ever changing business processes
- Difficulties retaining top talent to maintain code
What is really needed at the enterprise level is a centralized view of business processes that coordinates expert's solutions on each platform with the ability to seamlessly manage, troubleshoot and report across the enterprise. One option I would like to introduce is JAMS Enterprise Job Scheduling from MVP Systems Software, Inc. JAMS is a Windows based centralized Job Management Solution built on .NET that natively integrates with numerous platforms to address the needs outlined above. JAMS will enable your individual platform experts to build the necessary solutions then centralize and coordinate the business processes.
Application Integration with JAMS
The JAMS fully-featured graphical user interface enables you to develop, manage and report on your enterprise business processes in one application. The intuitive JAMS interface yields a low learning curve. There are a number of clients available to users, however the focus for this article will be on the Windows client, where team members can build new business processes calling code from across your enterprise. In addition, JAMS includes enterprise level security management, job monitoring, reporting and more.
Below is an example of the JAMS interface to monitor Jobs (on the right), with the navigation (on the top and left). With JAMS you can easily point and click through all of your enterprise jobs in a single view. This yields a significant time savings as compared to logging into numerous servers, reviewing numerous emails or parsing log files from each business process.
Figure 1 - JAMS User Interface
Coordinating Business Processes in JAMS
With the JAMS user interface, technology professionals can easily build business processes by dragging and dropping objects from the tool box to the work space. Once the object is in the work space, logic is added to the object. This enables subject matter experts to build workflow code for enterprise applications that is in the native language and most efficient for the platform, then coordinate all the business processes with JAMS. In addition, JAMS includes conditional and branching logic necessary for your business needs to conditionally start processes, handle exceptions, notify the team and more, as shown below.
Figure 2 - JAMS Workflow
As shown in Figure 2 and 3, the JAMS Toolbox features more than 75 objects to meet your development needs. Figure 2 outlines many of JAMS objects and Figure 3 highlights many of the native logical operators.
Figure 3 - JAMS Toolbox
Beyond the workflow shown in Figure 2, there is also flexibility at the JAMS job level to develop code directly to pass parameters between jobs, call secured credentials to access your enterprise applications, conditionally process files \ data and more as shown in Figure 4.
Figure 4 - Development with JAMS Jobs
With all of the development capabilities in JAMS, the JAMS team at knows that it is imperative to be able to manage your code. In the SQL Server industry, SQL Server Agent code management is a common challenge. For example, in SQL Server Agent you have the ability to change code, but there is not a record of the changes, who made the change, what code was changed and the ability to rollback code directly in SQL Server Agent. JAMS recognizes these needs as a development and enterprise job scheduling solution and provides solutions to these needs and more.
With JAMS centralized logic, the code is versioned in order to see the changes to code between two versions as shown in Figure 5.
Figure 5 - JAMS Code Comparison
After the specific versions of the code in JAMS have been reviewed as shown in Figure 5, you have the ability to review when code changes were made, by whom and the associated version. At that point, you can select a version of the JAMS code to rollback to at any point in time as shown in Figure 6. This Revision List provides a history, checks and balances as well as the ability to easily rolling back to a stable code base to support the business.
Figure 6 - JAMS Code Revision List
Security is top priority for enterprises to protect their customers and avoid the security breach publicity that has become a focus in the media. The JAMS team realizes the importance for enterprise level security and has made security a priority with features to support both enterprise applications and users.
First is the JAMS application level security which is used to connect to enterprise applications as shown in Figure 7. JAMS stores credentials in an encrypted manner and enables your development team to access an enterprise resource without having to know the credentials. There are also a number of configurations to protect these even more solidifying this security.
Figure 7 - JAMS Enterprise Application Credentials
The second layer of JAMS security is at the user level to enable your team to submit jobs, monitor jobs, review history, etc. all yielding significant time savings for your team. This removes the burden from your technology department and enables your users to run specific business processes with a self-service model based on your business needs. The JAMS user security is implemented at the Folder level where Jobs are logically organized. At the Folder level you have the ability to set security options such as starting, monitoring or stopping jobs then these permissions are inherited by all subfolders and jobs within that folder.
Figure 8 - JAMS Folder Security
Business Process Monitoring
With critical business processes that span numerous enterprise level applications, there are many moving parts and numerous challenges. In these situations, the Technology Team needs insight into the overall business process with the ability to drill into the details for troubleshooting, correction and the final resolution. JAMS is able to simplify the complexity in order to zero in on the issues across your enterprise, all the way down to the line of code. There are a number of interfaces in JAMS to address monitoring across the enterprise including customizable dashboards. Figure 9 is the JAMS interface for Monitoring Current Jobs. This interface is one of the options to view high level status but also analyze the details.
Figure 9 - JAMS Job Monitoring
For historical jobs, JAMS provides detailed level data in the Log File tab for a specific job as shown in Figure 10. The data includes when the job started, who initiated the job, credentials used as well as performance metrics (time, CPU, IO, memory, etc.), providing details that are often not captured but critical when troubleshooting a business process.
Figure 10 - JAMS Log File
On the other side of the coin, if you are unsure about what happened at a given point in time, JAMS provides an interface to query job history. JAMS retains all history of your jobs so you have the ability to search based on folder, job, time frame, user, etc. as shown in Figure 11.
Figure 11 - JAMS History Query
Troubleshooting a particular Job and searching historical data is very helpful. However there are many times when it is necessary to take a step back and understanding the context of a particular Job execution. For example, did the Job run within normal parameters or not? The JAMS team realizes that knowing the context of current Job execution is important and has included the Statistics interface to compare the minimum, average and maximum executions for five key counters (Elapsed time, CPU time, Direct I/O, Buffered I/O and Page Faults) to address this need as shown in Figure 12.
Figure 12 - JAMS Monitor Detail - Statistics
As a final monitoring item, JAMS includes numerous reports and the ability to customize reports and dashboards to best meet user needs. One item that is particularly interesting is the Aggregate Job History report that provides insight across your enterprise in terms of the successful and failed jobs in order to understand the norm and deviations from the norm to see enterprise-wide issues. This type of report enables your organization to understand overall trends allowing troubleshooting to most efficiently support the organization.
Figure 13 - JAMS Aggregate Job History Trends
Native JAMS Integration
JAMS natively integrates with more than 25 applications. This seamless integration by JAMS simplifies the development process and enables your team to focus on core business needs, saving significant time for development and day-to-day management.
Business Intelligence and BIG DATA
Figure 14 - JAMS Enterprise Application Integration
How do I get started with JAMS?
- Check out the free resources available for JAMS.
- Download JAMS to see how it can help you.
- Think about all of the challenges you face with Job Scheduling, Reporting, Notification, Workflows, etc. and how JAMS can solve these problems.
- Communicate with your team and management about how you think JAMS will
- Native integration and high speed data access to your heterogeneous enterprise applications
- Centralized troubleshooting, trending, reporting and management across your business processes
- Flexible security for data access to enterprise applications and simple security delegation in JAMS
- Time savings with a low learning curve to integrate JAMS into your enterprise
- Enable platform subject matter experts to work efficiently on the technology they know best then let JAMS centralize and coordinate business processes
- Put JAMS through its paces in your environment, share the results with your team and determine your next steps.
- We have just scratched the surface of the functionality available with JAMS. If you are facing data and application integration issues in your environment, consider JAMS as a solution to address your needs. Getting started is easy.
- Sound interesting? Check out these resources:
MSSQLTips.com Product Editorial sponsored by MVP Systems Software, Inc. makers of JAMS Enterprise Job Scheduling.
Last Updated: 2017-01-10
About the author
View all my tips