HSM : tips-n-tricks

A shameless copy from all over the place. Mainly for future reference and to be editted. Raw data if you like. Anyway : here's the copy paste :)

F HSM,SWAP LOG

F HSM,CANCEL USERID(userid)

F HSM,SETSYS etc

F HSM,CANCEL DATASETNAME(datasetname)

F HSM,CANCEL REQUEST(requestnumber)

F HSM,STOP

--

Dump for Installation Exit Abends

PATCH .MCVT.+2D BITS(.......1)

Tracing OPEN/CLOSE/EOV problems

Tape - PATCH .MCVT.+F2 x'00' /* Activate */
PATCH .MCVT.+F2 x'FF' /* Deactivate */
DASD - PATCH .MCVT.+F3 x'00' /* Activate */
PATCH .MCVT.+F3 x'FF' /* Deactivate */

Causing DUMP when DSS error occurs

PATCH .MCVT.+454'390' /* ADR390E */

Determining why SMS-managed datasets are not being processed

PATCH .MGCB.+26 x'FF' /* Space Mgt */
PATCH .BGCB. +24 x'FF' /* Backup */


--

HSEND LIST TTOC (A01086) ODS(list.output.filename)


--

HSEND QUERY ACTIVE

The common commands are

QUERY ACTIVE - see what processes are running, and the status of the HSM components
QUERY WAITING - see how many commands are queued waiting for resource
QUERY USER - list the users who are running active or queued commands
QUERY PROCESS - query active processes
QUERY SETSYS - see below
QUERY SETSYS is useful to see what the DFHSM settings are. Some of these are defined in your ARCCMDxx member in SYS1.PARMLIB, but they can be overridden by command, some system settings are set by a one-off command, and some take defaults. QUERY SETSYS will show you the list of all current system setting.

HOLD Command
You use this command to pause selected DFHSM function. However be aware that some holds can take a while to take effect, especially those that involve tape processing. A very useful command in a panic situation is

HOLD ALL

which stops all HSM processing. You will then probably want to start up selected functions using the release command.

HOLD MIGRATION
HOLD RECALL(TAPE)

Should both be self explanatory. You would want to hold tape recalls if you have a problem with your tape drives, but want recalls from ML1 to continue as normal.
The Query Active command above is useful to find out if any HSM functions are held.

RELEASE Command
If something is not working in DFHSM, you could try a QUERY ACTIVE command to check the status of that DFHSM function. If it is HELD, then you can release it with the RELEASE command. Of course, there may be a reason why it is held, so it is best to check this out first. The release command is simply

RELEASE function

You may want to use this in a panic situation as follows

HOLD ALL
RELEASE RECALL

This holds everything, then releases RECALL, which minimises user impact.
However note that if journalling has been disabled then RELEASE will not work until a CDS backup completes.

CANCEL Command
You use CANCEL to terminate an outstanding DFHSM request. You need to know the request number to do this. For example if a single request is causing problems then you need to find out the number of the problem request

QUERY REQUEST

ARC0101I QUERY REQUEST COMMAND STARTING ON HOST=1
ARC0162I RECALLING DATA SET file.name FOR USER user1, REQUEST
ARC0162I (CONT.) 00006399 ON HOST 1
-- more requests ----
ARC0101I QUERY REQUEST COMMAND COMPLETED ON HOST=1

You identify that the recall is causing problems, and the request number is 6399 (Actually you cannot cancel active requests, just queued requests)


CANCEL REQUEST(6399)

If a single user has a large number of requests in the queue then you may want to cancel all their processes to relieve congestion. The QUERY USER command will give you the userid, then you use the cancel command


CANCEL USER(userid)

You can cancel recalls by dataset name with the following command


CANCEL DSNAME(datasetname)

--
F DFHSM,BACKVOL CDS

F DFHSM, AUTOBACKUPSTART(0730 0900 1030)

F DFHSM,SETSYS SECONDARYSPMGMTSTART(0735 1030)

F DFHSM,SETSYS PRIMARYSPMGMTSTART(1035 1335)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Holidays
F DFHSM, DEFINE PRIMARYSPMGMTCYCLE (NYYYYYN CYCLESTARTDATE(2002/05/05))
F DFHSM, DEFINE SECONDARYSPMGMTCYCLE (NYYYYYN CYCLESTARTDATE(2002/05/05))
F DFHSM,DEFINE BACKUP(NYYYNNN 2 CYCLESTARTDATE(2002/05/05)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

DFHSM,QUERY SPACE

F DFHSM,BACKVOL CDS

F DFHSM,RELEASE MIGRATION

F DFHSM,RELEASE RECYCLE

F DFHSM,RELEASE ALL

F DFHSM,BACKLOG CDS(NULLJOURNALONLY)

F DFHSM,SETSYS PRIMARYSPMGMTSTART(1430 1730)

F DFHSM,Q CDS F DFHSM,HOLD LIST

F DFHSM,HOLD RECYCLE

F DFHSM,QUERY ACTIVE




F HSM1,HOLD ALL F HSM1,RELEASE ALL

F HSM1,HOLD AUTOMIGRATION

F HSM1,RELEASE AUTOMIGRATION

F DFHSM,QUERY [ACT, AUTOPROGRESS, CONTROLDATASETS, CSALIMITS, WAITING]

F DFHSM,QUERY [ABARS, STARTUP, SPACE, SETSYS] F DFHSM,STOP ALL

F DFHSM,HOLD ALL

F DFHSM,RELEASE ALL

F DFHSME,REPORT DAILY FUNCTION(RECOVER)

LIST TTOC SELECT(FAILEDRECYCLE) ODS(LIST.EXAMPLE.ONE)

DEFINE BACKUP(NYYYYNN 2 CYCLESTARTDATE(2002/05/05))




From the ISPF, option 6 panel

HSEND EXPIREBV DISPLAY ODS(userid.DISPLAY.LIST)

HSEND WAIT EXPIREBV EXECUTE

HSEND RECYCLE DISPLAY DAILY PERCENTVALID(15)

HSEND RECYCLE DISPLAY ML2 PERCENTVALID(15)

HSEND LIST VOL(W00340) TERM

HSEND LIST TTOC(W00340) TERM BOTH

HSEND LIST LEVEL(SP) BCDS SEL(AGE(400)) TERM

HSEND LIST DSN(DATASET.NAME) BOTH TERM


--


To be editted :)

HSM in a Virtual Tape Environment - Part 2

Yet another entry regarding HSM in a Virtual Tape Environment.

This time we will be adressing the impact of PtP implementations in backup and migrate (ML2) processing.

First of all, let's take a look at a timeline for creating a single-file-single-volume dataset on a PtP volume with copymode settings to imemdiate. In this case an HSM migrate level 2 dataset, but you this is also true for other single-file-single-volume datasets.





The data is first written at the local site. Before completing the O/C EOV (rewind unload) the volume is copied (in compressed format) to the remote VTS. Penalty for creating this 'synchronous' copy is the time required to perform this copy (on average between 40% and 60% of the 'original' job I/O time).

Now what happens if an HSM migrate task results in a dataset spanning multiple (lets assume 2) volumes?

At the point in time where HSM needs to get the second volume a rewind-unload is issued. Analogous to the first example this is followed (due to the immediate copy mode) by the copy to the remote site before this rewind-unload is completed. The time it takes to complete this copy causes a delay for the HSM task. As soon as the copy has completed, the second volume is mounted. At completion of writing to the ML2 dataset another rewind-unload will be issued, followed by another copy process to the remote site.

This results in the 'penalty' for creating the immediate copy is yet again roughly 40%-60% of I/O time.

For non-HSM multivolume datasets specifying unit=(,,2) will create an improvement in elapsetime because the PtP-copy can then be made in parallel to the local write to the second volume (see figure below). Unfortunately HSM allocates only one unit to each migrate tasks. Maybe we will see a SETSYS MAXCONCURRENTML2UNITS in the future giving us the option to fully exploit the VTS capabilities.

As reference a figure displaying the manner in which single-file-multiple-volume tape datasets are being created in a VTS-PtP environment with unit=(,,2) specified:

For the sake of completeness, all the three figures in one handy A4-sized picture. This image can be freely pinned in your cubicle.

Whadda-ya-mean-D/R?

Just so I will remember the link myself, but maybe interesting for the 1.5 (on average) reader of this blog : recoveryspecialties.com has some nice texts on PtP, PPRC and the such

Whadda-ya-mean-z/os?

For those of you who stumbled upon this site "by accident" but who are curious to the whole z/OS world, or maybe for the 3 regular readers the following link.

IBM just released a new redbook entitled IBM System z Strengths and Values, you can read the abstract (and even the complete redbook) right here : http://www.redbooks.ibm.com/abstracts/sg247333.html

It's not too technical, maybe even a bit too much commercial but good reading material nontheless.

REPORT DAILY FUNCTION

Ever had a wee look at the output from a 'HSEND REPORT DAILY FUNCTION ODS('YOUR.DATASET')' ?

I want to use todays post to rant a bit about the values presented here. As you see in the picture , I have boxed some of the values with a very nice red box :)

After reading my rant about these (fictive) figures feel free to leave your comments.

This might very well be the first in a series of 'how do I rate my figures'. So without further ado, here's my 2cc on the matter.....

Average Age. For the migrates and the recalls it's quite nice to be knowing what the average age is. Low values in the 'recall' rows might indicate HSM thrashing (recalls just after a migrate). The values in the 'migrate' rows must match your gut feeling in the average setting for migrates. (14 days Ml1, 30 days Ml2 or whatever your shops guidelines are).

Average Queued Time. Now this is an odd figure to be interpreting. It represents te amount of time an average request was held up in the queue. It's the average amount of time a request to recall a ML2 dataset was not served by HSM. Generally speaking: if it's high (> 120secs is my RoT atm) you've got more request coming in than HSM may/can handle. This might be due to SETSYS TAPEMAXRECALLTASKS, the available units or can even be due misuse :)

User Initiated Backups. Another nice figure to determine user behaviour. I've seen reports where every monday 400 userbackups were being made. User migrates are another thing to look at. Then it's the hunt for Red October, or whatever the user is doing these actions. The art of trying to find out why they are doing this, without assuming it's a wrong thing to be doing is a tricky but fine one. Remember, user migrates cause partial tapes!

Graphical View HSM Activity

Ever wanted to have a quick visual on your HSM activity?
By feeding the SYSLOG (or a selection of the SYSLOG as long as it has the ARC-messages) into the following SAS you get a nice report showing the times and durations of your HSM tasks.

The SAS assumes the SYSLOG file is a file spanning the period of no more than one 24 hour period.

How is this graph created?

Just downlaod the source : http://www.savefile.com/files/1528255

And there you have the complete source to craft your very own HSM activity Graphs!

Is the volume SMS managed?

To see if a volume is SMS managed, issue the following DEVSERV command

ds,s ccuu

Possible outcomes vor sane queries :

ccuu,xxxx ,F, , /RSDNT, VOLUME NOT MANAGED BY SMS
ccuu,xxxx ,F, , /RSDNT, VOLUME MANAGED BY SMS

Resize (or move) DfRMM Journal Dataset

Ever wanted to resize the RMM Journal dataset? Ever wanted to do this without stopping RMM?

This is the short recipe for doing so:

1. Allocate your new journaldataset (DFRMM.NEW.JOURNAL)
2. Quiesce DFRMM (f DFRMM,quiesce)
3. Run EDGHSKP, PARM=(BACKUP) to empty the Journal
4. Rename current journaldataset
5. Rename DFRMM.NEW.JOUNRAL to whatever is in your EDGRMMxx (or change EDGRMMxx)
6. Restart DFRMM (f DFRMM,m=xx)
7. Be a good boy and delete the old journaldataset

For calulating the required size of your journal dataset you can use the formula from the Implementation and Custumization Guide:

• 1.5 KB for each user change made
• 1.5 KB for each data set retained by a vital record specification
• 1.5 KB for each data set no longer retained by a vital record specification
• 1.5 KB for each expiring volume
• 6.7 KB for each specific mount
• 8.3 KB for each scratch mount
• 1.5 KB for each volume retained by a vital record specification
• 2.6 KB for each volume in or out of the library
• 3.0 KB for each volume returned to scratch
• 3.5 KB for each volume moved to or from a storage location
• 1.5 KB for each volume that has not reached its expiration date and is no longer retained by a vital record specification
• 1.5 KB for each logical volume exported or imported
• 1.3 KB for each vital record specification created

My RoT when resizeing the journal dataset is to just triple the size, as changes regarding the tape environment are mostly not reflected in the sizing parameters of the journaldataset :)

To make this a complete HOWTO, below you will find the JCL to backup the journal when DFRMM is not active.... (dots inserted for readability)

//STEP EXEC.. PGM=EDGBKUP,
//............PARM='BACKUP(DSS)'
//SYSPRINT DD SYSOUT=*
//MASTER...DD DISP=SHR,DSN=YOUR.CDS
//JOURNAL..DD DISP=SHR,DSN=YOUR.OLD.JOURNAL
//JRNLBKUP.DD DSN=YOUR.JRNLBKUP
//............DISP=(,CATLG),
//............AVGREC=U,SPACE=(9000,(500,500)),
//............LRECL=9000,RECFM=VB

HSM in a Virtual Tape Environment - Part 1

To have HSM create its (his? her?) backups, migates and dumps in a virtual tape enviroment some issues regarding tape recycling have to be addressed.

Because tape is virtual, the old-skool need for recycle (free up tapes!) is no longer as valid as it was, as the amount of tapes is a multitude of the amount one had in a physical environment. Virtual tapes however are not limitless, and one should keep the amount of tapes HSM 'manages' in repectable quantities.

Major drawbacks on the HSM recycle in a virtual environment is the immense negative impact this 'housekeeping-task' has on the cache on the virtual tape environment. This is even a bigger issue when one has more than only HSM as an exploiter of the VTS.

Ideally one would want to only recycle the empty tapes (percentvalid = 0). In order to get some isight in the the 'príce' one has to pay for this (via a higher tape consumtion by HSM) one should look at the following figures:

• number of tapes being used from the scratch pool each day
• how many of these are HSM
  
• life cycle for your non-hsm tape data
  
• life cycle for your hsm data (mostly dependent on life-cycle for original data)
  (some parameters influencing this are RETAINDAYS ONLY BACKUP, RETAINDAYS EXTRA BACKUPS, PRIMARY DAYS, LEVEL 1 DAYS)
• number of tapes being freed by (daily/weekly?) recycle tasks
• Valid Data Distribution on HSM datasets (pctvalid from LIST TTOC) (=trigger for recycle)
  
• Valid Data Distribution on HSM tapes (this is *not* pctvalid from TTOC) a tricky value in a VTS environment
  

The nest part of this post will address how all these values will provide the information needed to tweak HSM (and maybe even the VTS) to reduce the negative impact of HSM in a VTS as much as possible.

VTSSTATS in SAS

For years I have been working in a shop where VTSSTATS (from IBM) was the MO for creating reports for the PtP VTS. After swithching jobs I came to work in a place where raw smf data was not kept.

This was a shock. However the alternative was so much nicer ; SAS. At first I hated the bitch but within months I started to love her (cue Guns n Roses music lol).

Anyway. After some time of getting to understand the basics of SAS, and the power it unleashes I started to create a VTSSTATS based on SMF94 records in a SAS DB. For one it is awesome in creating graphs. One of the best things is the little amount of effort needed to create animated graphs.

As a cliffhanger so you'll be back to get the full source download, here is one of the animated graphs. For those of you familiar with the graphs, its an animated version of the Active Data Distribution.

The reports are quite similar to the ones I origingally used from the VTSSTATS and VTSANAL94.xls but here is the complete list of graphs created :

• 01-VirtualMountsD1.gif.


• 02-VirtualMountsD2.gif


• 03-VirtualMountsDailyD1.gif


• 04-VirtualMountsDailyD2.gif


• 05-CacheReadWritesC.gif


• 06-CacheReadWritesD1.gif


• 07-CacheReadWritesD2.gif


• 08-StackedVolumesC.gif


• 09-BackstoreCapacityD1.gif


• 10-BackstoreCapacityD2.gif


• 11-VirtualMountTimes.gif


• 12-VirtualMountTimesH.gif


• 13-CacheMisses.gif


• 14-TotalDailyPhysMounts.gif


• 15-TotalDailyPhysMountsD1.gif


• 16-TotalDailyPhysMountsD2.gif


• 17-AveragaDailyPhysMounts.gif


• 18-AveragaDailyPhysMountsD1.gif


• 19-AveragaDailyPhysMountsD2.gif


• 20-AveragaDailyPhysMountsD1.gif


• 21-AveragaDailyPhysMountsD2.gif


• 22-DailyAverageMissTime.gif


• 23-HourlyAverageMissTime.gif


• 24-TotalAccessorMounts.gif


• 25-AvgAccessorMounts.gif


• 26-MaxConcPhysMounts.gif


• 27-MaxConcVirtMounts.gif


• 28-AvgThrottlingDayD1.gif


• 29-AvgThrottlingHourD1.gif


• 30-WritThrottDayD1.gif


• 31-WriteThrottHourD1.gif


• 32-OverallThrottDayD1.gif


• 33-OverallThrottHourD1.gif


• 34-CachTimesHourlyD1.gif


• 35-CachTimesDailyD1.gif


• 36-CachTimesHourlyD2.gif


• 37-CachTimesDailyD2.gif


• 38-MountedTimeDC.gif


• 39-MountedTimeHC.gif


• 40-DataToBeCopied.gif


• 41-DataDistributionD1.gif


• 42-DataDistributionD2.gif
  

HSM LIST TTOC

Here's the challenge:

Take the output from one (or more) HSM LIST TTOCs and store them in a SAS PDB keeping date and time the TTOC as made.

For this example we will only store a limited number of fields from the HSL LIST TTOC output, I assume you can fill in the blanks if needed :)

Without further ado, here's the SAS code:

First, we need to parse the entire input file (yes this can be a GDG base of the GDG containing all your LIST TTOCS!) (blogger still dont do blanks for me, so code has been downgrades with spaces for readbility)

data hsm01;
.. infile hsmttoc;
.. input @31 selstring $16.
.... @75 theDate $8.
.... @63 theTime $8.
.... @95 theSyst $8.
.... @1 VOLSER $7.
.... @19 TYPE $7.
.... @46 PCTVALID $4.
.... @101 LIBNAM $9.;


Next we wat to 'save' the time and date from the LIST TTOC HEADER.

data hsm02;
.. set hsm01;
.. if selstring = "TAPE VOLUME TTOC" then do;
.... call symput('saveDate', thedate);
.... call symput('saveTime', thetime);
.... call symput('saveSyst', thesyst);
.. end;
.. theDate = symget('saveDate');
.. theTime = symget('saveTime');
.. theSyst = symget('saveSyst');

Now we can drop all the unwanted lines, I took the lines with the LIBNAME as valid lines. I assume you know what value to use for libnam ;)

data hsm03(drop= selstring libnam);
.. set hsm02;
.. where libnam = 'YOURLIBNAM';

And there we have it, a nice collection with all HSM owned volumes, their type (backup, migrates dumps) and their percentvalid.

Still to do, or why this problem arised in the first place, complete the SAS job so we will end up with a nice 'active data distribution'-graph.

This is something for next time :)