ibm_bpm

Agile and IBM BPM – Déjà vu

thaxtonmLeave a comment

I’m sure plenty of software engineers have been in this position: the rough surf of Agile and Scrum knowledge waves. Weekly blast emails about a new Agile garage or a new Product Owner organization change. Oh wait, Spotify a Guild? Let’s do it! Hire a Scrum Master – go, go go! Rollout DevOps – now!

It can be overwhelming.

But don’t get me wrong: I’m incredibly excited about the opportunity and the change and the excitement about Agile.

But I do have something small that I find a little amusing…I pulled the above image directly from Scrum.org and it reminded me so much of this old slide from an IBM BPM sales presentation about iterative development. I no longer have an exact copy of the deck, but check out this image from Page 11 of a 2015 Redbook:

2015 IBM BPM Design Redbook

And I know: Agile and Scrum have been around for years, I get that. But in some organizations, even in 2021, it is all brand new. But as an IBM BPM team we’ve been practicing this iterative development pattern for years, even in Waterfall projects, right?

Check out this Bruce Silver article about IBM BPM’s predecessor (Lombardi TeamWorks) in 2006:

A second distinguishing characteristic is support for rapid iterative development. Other vendors frequently pay lip service to this implementation style, but Lombardi supports it concretely with the ability to instantly “play back” activities and process fragments even early in the modeling phase, creating default components as necessary that will be refined later on. Lombardi emphasizes a project methodology in which a simple version or fragment of the process is piloted very quickly, with additional features and richness layered on iteratively after that.

Bruce Silver 2006: https://cs.nyu.edu/~jcf/classes/CSCI-GA.2440-001_sp12/handouts/BPMS_-_Lombardi_01.pdf

“Play Backs” and “rapid iterative development” – how much more Agile can you be?

It’s so nice to see other application teams excited (well, excited and full of trepidations) about Agile. Where they traditionally waited around for complete technical specification documents and design documents, now they can start coding and testing at the same time the larger team is building complementary code and functionality for stories. Where they were heads-down in code until QA, they are heads-up with the team each day, refining stories, providing Dev feedback, engaging the business, validating acceptance criteria, etc.

It’s seem like such “old news” to BPM developers, but it’s very exciting to have more partners along for the ride going forward.

Another IBM BPM/BAW Date Hack

thaxtonmLeave a comment

The team had a user story to determine the Thursday of the 2nd week of the first month of a quarter (so January, April, July and October), recognizing that a week might only contain one day of the month. This would be server-side code running in a BAW Service Flow asset.

January 2021 is a good example:

Notice that January 1 and 2 fall on Friday and Saturday but our user story considers that a valid “week” for the month. So the first Thursday of the 2nd “week” would be January 7.

We had existing code from a prior version of this requirement where the user story was written as “the 2nd Thursday of the Month”, which would end up on January 14, 2021 because we used a method to get the first occurrence of the weekday (Thursday is getDay() == 4 in this case) of the month using mod (%) 7 then moving that date of the month (1-31) out by 7 until we landed on the correct 2nd occurrence: (2-1)*7.

With the new user story, we still determine the Thursday of the 2nd week (that code has been successfully tested), but then we check to see if that week is actually the 2nd week in the month. If not, we subtract 7 days and use that value.

Here’s the new code:

var c = new java.util.GregorianCalendar.getInstance();
c.set(tw.local.targetDate.getFullYear(), tw.local.targetDate.getMonth(), tw.local.targetDate.getDate());
c.setMinimalDaysInFirstWeek(1);
var wk = c.get(java.util.Calendar.WEEK_OF_MONTH);

if (wk > 2) {
   // If we're not in the 2nd week, move back one week (7 days)
   tw.local.targetDate.setDate(tw.local.targetDate.getDate() - 7);
}

We start by creating an instance of the Java GregorianCalendar (the regular Calendar option doesn’t seem to work in either BPM, Java 8, or Rhino, for some reason). And we set the calendar to the matching year/month/date of targetDate (this was previously defined as the 2nd Thursday of the first month in the quarter using the existing code).

The we call setMinimalDaysInFirstWeek to define how many days we consider to be in the first week of the year (don’t worry, this carries over to the rest of the calendar year – October 2021 has the same setup as January 2021). In this case – just 1 day is considered a “week”.

Then we use the WEEK_OF_MONTH constant to determine which week of the month we’re in and decide if we can keep the current targetDate or set it back 7 days to the previous week (it will never be more than 3 weeks off because of the existing “2nd Thursday of the month” logic).

The GregorianCalendar object was helpful and being able to merge the Java and JavaScript was much easier than trying to transcribe into JavaScript or write something entirely from scratch.

The Java GregorianCalendar has a few other helpful methods, one of which is “roll”, but it behaves differently in Java 7 vs Java 8, so consider your environment if you want to experiment with this object.

The 12 Factors of IBM BPM

thaxtonmLeave a comment

So I was taking some Agile training the other day and came across this great resource I had not seen before: https://12factor.net/

That list provides a methodology to help architect and design applications that primarily provide services, not necessarily UX or front-end solutions.

IBM BPM is a unique product because it can provide full-stack solutions with process flows and Coaches, but it can also function solely as a service application (think of a headless process app where client applications like UX interact with the BPM REST API).

So can any (or all) of those 12 Factors apply to IBM BPM process applications? Let’s take them one-by-one and give me a Match or Unmatch verdict.

  1. Codebase
    This one seems pretty easy: IBM BPM includes Process/Workflow Center as a central design-time resource and asset repository…but then again, this one is tough because IBM BPM doesn’t really integrate with common source-control systems, like Git or VSS or SVN. I’ve seen lightly-coupled integrations; for example: use an automated process to kick-up off the install ZIP process and store the file in Source Control, then deploy from there using another process. But Factor #1 mentions tracking…this is the tough part. Everyone knows how easy it is to track changes in most Source Control, especially something as simple as Git. Process/Workflow Center on the other hand isn’t very consistent with change tracking. I think Desktop PD was better than WebPD, by far. So maybe the time stamps and user IDs are hidden in the DB somewhere, but they aren’t always on the WebPD UI.
    Verdict: Partial Match
  2. Dependencies
    I read this as Toolkits, right? Toolkits are really, really helpful in IBM BPM and create easily reusable assets that can shave off tons of development time, especially if the TKs are properly managed (think dependency updates and dealing with nested TK references).
    Toolkits can also introduce a lot of headache if they constantly reference each other over and over.
    I’ve used the BP3 Dependency Checker for years (still running the 7.5 version) and it’s one of the most helpful ways to track TK relationships and avoid potential deployment issues with overlapping TK snapshots. I just wish the Process App and Snapshot dropdowns were alphabetized! 🙂
    Verdict: Match
  3. Config
    I see Environment Variables and Exposed Process Values as part of the Config, right? I think these two features of IBM BPM provide a massive amount of value to deployed process applications. Are they easy to maintain – that’s a different argument. But being able to configure run-time parameters via Process Admin is a great resource for processes that have fluid business rules.
    Verdict: Match
  4. Backing services
    I think this aligns with Toolkits…? Especially if you practice building toolkits for specific integrations. For example: you have an enterprise service that already exists for customer address information. There’s no need to build that SOAP or REST integration in each process app, so wrap the call in a Toolkit and manage the integration in a central location, then attach the resource to your app when and where you need it.
    Verdict: Match
  5. Build, release, run
    I think IBM BPM’s snapshot paradigm aligns well with the separation of these actions. We build the snapshot in Process/Workflow Center and deploy to a runtime Process Server. It’s not like we’re actively developing in a runtime environment. Now granted P/W Center has a Process Server inside of it, but that is mainly for unit testing and debugging.
    Verdict: Match
  6. Processes
    Though we’re building Process Automation Applications, I think the code and resulting process instances themselves are perfectly capable of standing alone as separate entities, but obviously they aren’t entirely “stateless”; especially since state persistence is one of the primary functions of a process/workflow solution. But couple this factor with IBM BPM running on WebSphere Application Server and the platform is executing each activity and service as their own threads in the WAS pool.
    Coming from another angle: are you designing your process at a level that can be re-used or incorporated in places other than its original target? For example: say you build a process application with Coaches to automate the hiring process: enter applicant details, route for review, post to SOR, etc. What if your company gets a new employee or HR portal but you want to re-use your existing process asset? Was your process designed to easily integrate without a Submit Coach? Or a Review Coach? Or a change to the SOR? It’s not that the process is stateless, but can your process application exist without some dependencies or easily adapt to changes in dependencies…?
    Verdict: Match
  7. Port binding
    I’m not sure this one is entirely relevant to an IBM BPM Process Application…Again, I think this is handled by WAS under the covers.
    Verdict: N/A
  8. Concurrency
    Another “covered by WAS” factor? But also a design consideration…are you designing your process assets so that you can add volume or functionality later without a complete re-write of the application? Can your process instances execute at the same time? Most process and workflow solutions probably have this requirement, but then there are others that need relationships between instances. I’ve personally never used this feature in IBM BPM but maybe there are some good use cases.
    Verdict: Match
  9. Disposability
    I’ll have to come back to this one…
    Verdict: N/A
  10. Dev/prod parity
    I think this one is easy if you isolate it to IBM BPM process applications. Especially if you have a pipeline that deploys your install ZIP to multiple non-prod regions (if you have them).
    The tough part comes when you introduce external dependencies, especially test data. A lot of that might be out of your control as a process application engineer. Sure you can mock data in P/W Center or maybe even in another non-prod region, but what if your service providers can’t do that or have different data sources for each region? This can make testing difficult, especially if your process logic needs to handle lots of different parameters from external integrations.
    Verdict: Match
  11. Logs
    I’m going to add a separate post for logs and log4j in IBM BPM, but I think this one is pretty easy to cover using log.* and the default WAS SystemOut. How you manage logs and when you use them are a larger topic for a longer post. Look for that coming soon.
    Verdict: Match
  12. Admin processes
    Again, you can use WAS and the Integrated Console or wsadmin for this factor, but don’t forget about the power of Process Admin. This runtime console is great for managing installed apps, has the utilities for ENVs and EPVs as well as some instrumentation/performance monitoring. That last point is sometimes really helpful or really frustrating, especially if you have a clustered ND setup with multiple servers. Unless you have a way specifically hit a server, this tool can be tough to use with a load balancer or other web server in front of IBM BPM. But either way Process Admin is still an extremely helpful tool for managing your runtime environment.
    Verdict: Match

So can IBM BPM be a 12 Factor app? I think in most cases it is a great match. But there is also a lot of variability in how you design your process flows and services to make sure you meet some of the factors. Especially when it comes to how to integrate with other applications and how your process starts and ends.

Remember that modeling a process in discovery and building a process for execution are often (if not 100% of time) two different models. So take time to plan the automation model so that it can incorporate as many of these 12 Factors as possible.

IBM Process Federation Server (PFS)

thaxtonmLeave a comment

In an older role I helped work on a proof-of-concept (POC) for IBM Process Federation Sever (PFS). At the time this was a really new offering from IBM in the BPM space. PFS is a WebSphere Liberty Profile and Elasticsearch-based source for BPM Task data.

IBM BPM already provided a way to federate instance and task data between two different BPM systems running the same version: essentially you register the systems with each other and one API call for instance or task data can pull the data from both systems.

PFS introduced a way to provide this same functionality to systems not on the same version. So think about running an IBM BPM v8.0 platform but you want to upgrade to v8.6. You could do that in place and put your platform at risk and require a lot of testing; or you could stand-up a v8.6 system and migrate apps one at a time to the new platform. But what if you don’t want your users to have to switch back and forth between the different Process Portals? This is where PFS comes into play: create a central source for task data that can be accessed by Process Portal (or any other work mgmt UX using PFS REST APIs which look just like the Federated BPM REST APIs) and that source can serve both the v8.0 data and the v8.5 data. Yay – win win!

PFS was not a performance solve, per IBM. It was just a way to consolidate task data in one location; the fact that they chose Liberty and ElasticSearch is totally up to them…both are relatively lightweight, so I’m sure that had something to do with it. I’m pretty sure Liberty is on the horizon for the BPMN engine so maybe it was their own little POC of BPM in Liberty.

Either way, when a BPM Task is created a bunch of stuff happens in BPM and the database, but most obviously a row is added to the LSW_TASK table. When someone logs into the Responsive Process Portal, a legacy* REST API call is made against BPM and that ends up hitting this LSW_TASK table (and a bunch of other tables) to determine if the user should see that task and be able to claim it and action on it. (*) By legacy I mean the existing IBM BPM REST API that has been with the product since Lombardi Teamworks 7.5.1.

A lot of other DB activity hits LSW_TASK, like when a task is created and actioned-on and claimed and even some instance-related events can hit LSW_TASK. The bottom line is that pretty much all IBM BPM developers and administrators realize that database latency should be as low as possible; the tech sales pitch was always “put BPM and its database right beside each other.”

Contrary to IBM’s standpoint, we started the PFS POC looking for a task source that displayed better performance then hitting the regular BPM REST API. In the POC environment the database resources were all centrally managed at the enterprise level. And though our DB and schema were in the same data center as our application servers, we still saw some latency between BPM and the DB. Attribute it to whatever you want…we were running Oracle 12c on Exadata as a two (2) node cluster. Do all the research you want – that’s a fast setup. But also recognize that we were one app in an incredibly large landscape of DB consumers. And most schemas were partitioned off with default settings and such. But as the BPM team, we weren’t DBAs and we relied on the DBA teams to manage that for us, which was totally fine.

So we thought “hey, a noSQL file-based BPM task repository with an existing API – yay speed!”

I’m not going tinto the details of our POC but in the end I learned a lot about how PFS works.

  1. First you have to build new tables in the BPM database for PFS information
  2. Second you have to configure BPM to use those new tables. How you might wonder?
    1. You essentially setup a trigger that says anytime BPM creates a task or does anything human-task related add a row to one of the PFS tables
  3. PFS is configured to regularly ping those BPM-schema PFS tables over JDBC to look for changes in the data – essentially PFS is looking for new tasks or changes to existing tasks
  4. Then PFS queries into the detailed LSW tables to collect the full task and business data
  5. PFS indexes that data into its Elasticsearch cache and updates the BPM-schema PFS table to flag the row as already read
  6. And this happens over and over as tasks are created and updated

NOTE: PFS does not have its own SQL database except for one to hold Saved Searches, which are defined over a REST API operation and can be used to search the task index in Elasticsearch

If you paid attention to that extremely simplified walk-through, you’d notice how many times I mention DB calls…so maybe you see? In an environment where we already had DB latency, why would we want more activity on the DB just to outsource the task data?

The POC still continues and there is a chance the new DEF might end up playing a role in the solution. But either way – just like IBM said, we didn’t find PFS to be a performance solve.

IBM BPM – EPVs

thaxtonmLeave a comment

IBM BPM provides a capability called Exposed Process Variables (EPVs). These assets allow you to use variables that can be changed in the run-time without a new code deployment. IBM BPM has a basic change utility and audit tracker for these values so you can use an out-of-the-box utility to see the current values, update them and also track when they were last changed and who changed them.

EPVs are great for parameters that might need to be changed frequently after a process application is deployed. I use EPVs a lot for things like threshold values or service integration parameters that are used for identification but not business logic.

Another helpful option is to define a comma-delimited list of strings that we might want to validate against a service response. For example if we call a service that returns a list of securities like IBM, AMZN, BLK, F, GPS but we only care about IBM and BLK for example, we can store the entities we care about in an EPV and iterate through the response list to see if the current item is one of our EPV values.

Use the code below as an example for parsing a comma-delimited EPV into a native JS array and then comparing the current array value to some local variable (tw.local.checkVar is this example)

// Get the EPV String
var currentEPV = String(tw.epv.EPVGroup.specificEPV);

          
if (currentEPV && currentEPV.length > 0) {
      // Split the EPV String into a native JS Array
      var currentEPVArray = currentEPV.split(/,/g);             
      for (var i = 0; i < currentEPVArray.length; i++) {
        var curItem =  currentEPVArray[i].trim();
        if (tw.local.checkVar == curItem) {
           // Apply whatever variable or logic check 
           // you need against the current value
           tw.local.logicFlag = true;                            
        }
      }
}

Installing IBM Process Designer (Desktop)

thaxtonmLeave a comment

For folks that still cling to the old ways, I’ve run into a lot of different scenarios when trying to install the Desktop IBM Process Designer.  I finally put together some notes for a way that works if you don’t have admin rights to your machine and the default install scripts aren’t working either.  Maybe this will help some folks that find it.

Navigate to Process Center (whatever your PC URL is…maybe something like):
https://pc_host:port/ProcessCenter

  1. Log on with your current Process Designer credentials
  2. Click on the “Download Process Designer” link in the right-hand navigation window or the pop-up window
  3. Save “IBM Process Designer.zip” to your local machine (~800 MB)
  4. Once downloaded, Extract the contents of the zip file
  5. If you don’t already have IBM Installation Manager installed:
    1. Of the extracted files, navigate to the IM64 folder
    2. Run the “userinst.exe” command to install IBM Installation Manager.
      NOTE: Be sure to install this program in a location accessible for your ID (eg – C:\Users\myid123\IBM\IM)
      You do not need to create the directory before running the installation
      The installation path must not exceed 40 characters.  The installation path should not containspaces
  6. Once the setup finishes it will ask you to restart Installation Manager
  7. Open IBM Installation Manager again (or let it re-open itself, whichever you need) and click File > Preferences and select Add Repository…
  8. Navigate to the folder where you extracted the ZIP file and go into the IMPD85 folder and select the repository.config file
  9. Click OK on the Preferences window
  10. Back on the Installation Manager launch page select Install
  11. Installation Manager will read the repository preferences then the config file and present you with the option to install IBM Process Designer > Version 8.5.X
  12. Check the box and click Next to select an Installation Directory
    NOTE: If you already have a version of Process Designer installed, Installation Manager will warn you of an existing package.  
    Just click “Continue” to install Process Designer to a new Installation Manager package group.
  13. Click Next and set the Installation Directory
    NOTE: again, be sure the directory is in a path that is accessible to your ID (eg – C:\Users\myid123\IBM\PD85x)
    You do not need to create the directory before running the installation.  The installation path must not exceed 40 characters.  The installation path should not contain spaces.  It is helpful to add the full Process Designer version to the installation directory (such as /PD857).
  14. Wait for Installation Manager to complete the installation
  15. Navigate to your installation directory (eg – C:\Users\myid123\IBM\PD856) and open the “eclipse.ini” file in a text editor
  16. Edit the line for the Process Center URL to correspond to your appropriate Process Center:
    -Dcom.ibm.bpm.processcenter.url=[INSERT PROCESS CENTER URL]
  17. Save the “eclipse.ini” file

All done.  Now you can open Process Designer

If you ever have the need to run multiple Desktop IBM Process Designer for more than one Process Center, use this:

Running the same version of Process Designer with multiple Process Centers

If you use more than one Process Center and the versions of each instance are the same, you can use one installation of Process Designer to access either Process Center.

  1. Install Process Designer from one of the Process Centers you use for BPM development
  2. Locate your Process Designer installation directory and find the eclipse.ini file
  3. Copy this file and add a suffix to designate the INI is for the other Process Center.  For example: eclipse_SANDBOX.ini
  4. Open the copy/renamed INI in a text editor and edit the line for the Process Center URL to correspond to your appropriate Process Center:
    -Dcom.ibm.bpm.processcenter.url=[INSERT 2ND PROCESS CENTER URL]
  5. Now find a shortcut to Process Designer (you can use the one in the Windows Start Menu)
  6. Rename the shortcut to include the name of the 2nd Process Center.  Like “IBM Process Designer Sandbox”
  7. Right-click on the 2nd shortcut and select Properties
  8. Update the Target field by adding –launcher.ini and the name of your 2nd eclipse.ini file.  For example: \path_to_PD\eclipse.ini –launcher.ini eclipse_SANDBOX.ini

IBM BPM – Date Diff Code

thaxtonm1 Comment

Here’s a few blocks of code samples that I like to reference when trying to deal with Dates in IBM BPM.

This first block (Date Difference Calculation Testing) helps you determine an integer number of dates between two TWDate variables.

// Date Difference Calculation Testing
// Date1 + xNumberOfDays = Today
// xNumberOfDays = Today - Date1
 
tw.local.aDate = new TWDate(); //SETUP SAMPLE
tw.local.aDate.parse("05/15/2005","MM/dd/yyyy"); //SETUP SAMPLE
 
if (tw.local.aDate && tw.local.aDate < new TWDate()) {
    // Today...as in right now
    var tday = new Date();
    
    // convert to native Java Date
    var aDay = tw.local.aDate.toNativeDate(); 
    var one_day = 1000*60*60*24; // milliseconds in 1 day
    
    var tm = tday.getTime(); // milliseconds for date1
    var am = aDay.getTime(); // milliseconds for date2
    
    var diff = tm - am; // milliseconds difference
    
    tw.local.dateInt = diff; // this is actually a decimal
    
    // convert milliseconds to days
    var days = Math.round(Math.abs(diff/one_day)); 
    
    // Integer of number of days between aDate and today
    tw.local.dateIntTwo = days; 
}

This is a cool way to tell if a date variable is within a specific threshold.  My use case was that the application stored when it made a specific integration call and then if we came back and needed that call again we’d check to see when it was last invoked.  This could easily be done with caching or something fancier but for this scenario a simple “was the last call too long ago?” bit of logic was all that we needed.

// Assume the service needs to be called again
tw.local.refreshNeeded = true;
 
// aDate is the value of the last time the service was called
if (tw.local.aDate) {
    var lastCall = tw.local.aDate.toNativeDate();
    var now = new Date();
    // EPV to store threshold in seconds to consider time stamp stale
    var threshold = Number(tw.epv.refreshThreshold);
    var thresholdMS = threshold * 1000;
    var lcMS = lastCall.getTime();
    var nowMS = now.getTime();
    var diff = (nowMS - lcMS);
    if (diff <= thresholdMS) {
        tw.local.refreshNeeded = false;
    } 
}