Diagnose and solve license problems

If a problem occurs, please track the following information, which can be useful to service personnel.

Which operating system is in use, and which version of that operating system.
Type of computer used as license server.
FLEXlm version. Use the lmver or lmgrd -v commands to identify the version. See lmutil for more detail.
Error or warning messages that appear.
Did the server start correctly?
Are you using a combined or a separate license?
Are you using a single server or multiple server redundancy?
What information is returned when you execute the lmstat - acommand? lmstat returns license management status information. See lmutil for more detail.
What information is returned when you execute the lmdiag command. lmdiag returns information relevant to license checkout problems. See lmutil for more detail.

FLEXlm license error messages

FLEXlm and FLEXnet provide error messages that can help guide you in diagnosing a problem. FLEXlm and FLEXnet errors with the same numerical code are identical.

For a complete listing of messages you will see in the "debug log" see Appendix G of the FLEXlm End User Guide.
(You can find the guide at the Advantest Web Site: Advantest Home > Global Services > Product Support > V93000 > V93000 Licensing > FLEXIm License Server Software Downloads and Documents > FLEXlm End Users Guide.)

**FLEXlm license error messages**
Error Code	FLEXlm License Error Description
-1	Cannot find license file.
-2	Invalid license file syntax.
-3	No server for this feature.
-4	Licensed number of users already reached.
-5	No such feature exists.
-8	Invalid (inconsistent) license key or signature. The license key/signature and data for the feature do not match. This usually happens when a license file has been altered.
-9	Invalid host. The hostid of this system does not match the hostid specified in the license file.
-10	Feature has expired.
-11	Invalid date format in license file.
-13	No SERVER lines in license file.
-14	Cannot find SERVER hostname in network database. The lookup for the hostname on the SERVER line in then license file failed. This often happens when the host file, or NIS, or DNS is incorrect. Workaround: Use IP address (for example, 123.456.789.123) instead
-15	Cannot connect to license server. The server (lmgrd) has not been started yet, or the wrong port@host or license file is being used
-16	Cannot read data from license server
-18	License server does not support this feature
-21	Reports FLEXlm license version
-26	Request for more licenses than this feature supports

The FLEXnet End User Guide at the Advantest Web Site: Advantest Home > (Support & Services) V93000 SOC > Software Downloads and Utilities for the V93000 SOC Series > FLEXIm License Server Software Downloads > FLEXnet End Users Guide.

Losing connection to license server

In case you loose connection to the license server (for example, due to a network outage) the software will continue working for two hours. After that time it will shutdown automatically.

A window (see below) is displayed every 30 minutes to inform you about the lost connection and the remaining time for which the software can still be used.

Changing the message interval

To change the interval in which warning messages are shown you can modify the flexlm.cfgfile in the /etc/opt/hp93000/soc_commondirectory. The entry WarningInterval <minutes> defines the interval in which warning messages are shown. Default value is 30 minutes.

License server behavior

If the connection is lost the license server frees checked out licenses after 20 minutes. This is the minimum time. You can increase the duration in an option file on the license server. The option file must be named socbu.opt and reside in the same directory as the vendor daemon socbu. For details on the socbu.opt option file please refer to the FLEXlm End User Guide.

When the license server connection is restored, the 2-hours-timeout is reset and licenses are checked out again.

Using the FLEXlm debug log

In case of problems with the license server, the status and error messages of the FLEXlm debug log will be useful.

The license server keeps its log files in the /opt/flexlm/log directory. All files are tagged with a time stamp marking the date of creation. The most recent file is available via a hard link: lmgrd_socbu.log. The last entries at the end of /opt/flexlm/log/lmgrd_socbu.log provide information about recent licensing issues.

The log files contain information about:

configuration problem messages,
daemon software messages, and
informational messages.

These messages are useful in case you need to debug the license server.

For a listing of possible debug log messages see the FLEXlm End User Guide.

You can find the FLEXlm End User Guide at the Advantest Web Site:
Service & Support > IC Test Systems > Licensing > V93000 FLEXlm License Server Software Downloads and Documents.

FLEXLM_DIAGNOSTICS environmental variable

The FLEXLM_DIAGNOSTICS environmental variable is mainly used to diagnose license check out problems.

Note: To diagnose license check out problems, Advantest recommends to set the variable to FLEXLM_DIAGNOSTICS=stdout instead of level 3. This provides the same data but avoids the creation of empty flex*.log files in your SmarTest start directory.

This diagnostic tool provides up to three levels of diagnostic, depending on the version of FLEXlm installed on your system. Each level increases the diagnostic functionality available when FLEXLM_DIAGNOSTICS is used. Level 2 includes functionality of levels 2 and 1. Level 3 includes the functionality of levels 3, 2 and 1.

The examples below display the information that will be retrieved when the environmental variable is set to 1, 2, or 3. Text in bold font indicates additional information provided by each level, compared to the preceding level.

Level 1:
For use when a license checkout fails to load. When set to 1 a FLEXlm licensing check out error message is shown, as well as a complete list of the license files that the application attempted to use.
Level 2:
For use when a license checkout fails to load. In addition to level 1 information, the level 2 diagnostic file returns information about the present checkout arguments. Among arguments listed in lm_checkout: feature, version, num_lic, queue_flag, dupgroup_mask.
Level 3:
For use both if a checkout is successful and if it fails to load. Information about how the license is granted is displayed; for example, whether a server is used, the name of the license file, and how the license server has been designated.

In case of a successful checkout, the log file is still created, but contains no information. We recommend to use FLEXLM_DIAGNOSTICS=stdout instead (see Note above).

Note: Default FNP license directory (path): /opt/flexlm/licenses

Example of Level 1

export FLEXLM_DIAGNOSTICS=1

FLEXlm checkout error: Cannot find license file (-1,73:2) No such file or directory license file(s): /path/licenses/testing.lic license.lic

Example of Level 2

export FLEXLM_DIAGNOSTICS=2

FLEXlm checkout error: Cannot find license file (-1,73:2) No such file or directory license file(s): /path/licenses/testing.lic license.lic

lm_checkout ("f1", 1.0, 1, 0x0, ..., 0x4000)

Example of Level 3

export FLEXLM_DIAGNOSTICS=3

FLEXlm checkout error: Cannot find license file (-1,73:2) No such file or directory license file(s): /path/licenses/testing.lic license.lic

lm_checkout ("f1", 1.0, 1, 0x0, ..., 0x4000)

app1

Checkout succeeded: f0/14263EAEA8E0

License file: ./servtest.lic

No server used

app2

Checkout succeeded: f1/BC64A7B120AE

License file: @localhost

License server: @localhost

app3

Checkout succeeded: f1/BC64A7B120AE

License file: ./servtest.lic

License server: @speedy

Note: Text in bold indicates diagnostic information returned for that level of diagnostic code that is not returned at a lower level.

Server and network crash diagnostic table

In any server or network crash scenario the test systems will continue to run the current tests for two hours or until there is a change of test device. System calibration and diagnostics will, similarly, result in stopping the current device test.

Test systems cannot initiate new tests until the failure case is repaired.

The Recovery column indicates the repair for each crash scenario. "ELF" in the table refers to the Emergency License Facility, that is explained in the section Emergency licenses.

Failure Case	Impact on Test System	Recovery
FLEXlm License Server Crash	Systems that get their licenses from this server stop once they change device, reboot, calibrate, run diagnostic. Otherwise the testers keep running	If Harddisk failure, repair Harddisk, install SW and existing licenses. If Motherboard failure, use ELF process via WEB to obtain 10 days license. Within 10 days obtain ReHosted licenses from Advantest Support.
Test Floor Network down	Systems that get their licenses from this server stop once they change device, reboot, calibrate, run diagnostic. Otherwise the testers keep running	For long time Network outage: Use Elf process on a network outside the test floor (Internet Cafe) to obtain local licenses for each tester computer. (gives 10 days access)
FLEXlm License Server Crash and Test Floor Network down	Systems that get their licenses from this server stop once they change device, reboot, calibrate, run diagnostic. Otherwise the testers keep running	Use ELF process on a Network outside Test Floor (for example, Internet Cafe) and obtain license. Use floppy disk, Flash card, any removable media, or print out license key and type in on test floor computers.
FLEXlm License Server Crash and Advantest Server (ELF) down	Systems that get their licenses from this server stop once they change device, reboot, calibrate, run diagnostic. Otherwise the testers keep running	No recovery until Advantest server is up again. (We estimate the probability of both failures occurring simultaneously at about 0.001.)

Mismatch in license server version

After having updated SmarTest you may get an error message which asks you to upgrade the license server:

Mismatch in License Server version (check <version>). 
License Server version <version> is older than required version <version>.
License Server upgrade is required, as otherwise no licenses can be checked out.

If you get this error message, you must upgrade the FLEXlm license server.

The procedure to upgrade the license server differs depending on the operating system of your license server host: Linux or Sun Solaris.

Upgrading FLEXlm executable files

This topic explains how to upgrade the license server using the FLEXIm executable files.

About this task

Upgrading the license server means updating these FLEXlm binary executable files:

lmgrd: the license server executable
lmutil: the license server utility
socbu: the license vendor daemon

After having updated SmarTest you may get an error message which asks you to upgrade the license server:

Mismatch in License Server version (check <version>). License Server version <version> is older than required version <version>.
License Server upgrade is required, as otherwise no licenses can be checked out.

The procedure to upgrade the FLEXlm binary executable files differs depending on the operating system of your license server host: Linux or Sun Solaris. In any case you must have root (superuser) rights to upgrade the binary files.

Alternatively to the procedure described below, you can reinstall the complete FLEXlm License Server RPM package.

Note: If you have a three-redundancy-license-server setup you must do the upgrade on all three license servers.

Before you begin

You must have downloaded the appropriate upgrade package, either a zip file or a tar.gz file, for your operating system from FLEXlm License Server Software Downloads and Documents on https://www.advantest.com/service-support/ic-test-systems/products-list/v93000-licensing. The following packages are provided:

FLEXlm License Server Components <version> for RHEL 5 and RHEL 7 64-bit (zip file)
FLEXlm License Server Components <version> for Sun Solaris 10 (tar.gz file)

Each of these packages contains the FLEXlm binary files lmgrd, lmutil, and socbu.

Procedure

To upgrade the FLEXlm binary files, login as root (superuser).
Open a terminal window.
The license server must not run. To check if the license server process is running, type
ps -ef | grep lmgrd

If the process is running, you will get an output for /opt/flexlm/bin/lmgrd and socbu like:
```
root  1234     1 0  11:11 pts/1  00:00:00   /opt/flexlm/bin/lmgrd ...
root  1235  1234 0  11:11 ?      00:00:00   socbu -T ...
root  1239  9480 0  11:29 pts/1  00:00:00   grep lmgrd
```
If the license process is running, stop the license processes. To stop the processes, type:
- RHEL 7:
  To stop the license server, log in as root user and use the following command:
  
  systemctl stop v93-license
- RHEL 5:
  To stop your license server use the smartest_license script:
  
  /etc/init.d/smartest_license stop
To extract the FLEXlm binary files from the upgrade package,
- on Linux, type unzipfilename.zip
- on Sun Solaris, type tar -xzvffilename.tar.gz
  This extracts the FLEXlm binary files lmgrd, lmutil, and socbu.
Copy the files lmgrd, lmutil, and socbu to /opt/flexlm/bin, replacing the existing version of the files.
If you have a three-redundancy-license-server setup you must do this on all three license servers.
Make sure the permissions for each file are set correctly. The files must be executable for the appropriate user, who will start the license server.
To start the lmgrd license process, type:
- RHEL 7:
  To start the license server, log in as root user and use the following command:
  
  systemctl start v93-license
  
  Note: The /opt/flexlm/bin/lmgrd [-c license_file_path][-l debug_log_path] command for starting the license server must not be used on RHEL 7.
- RHEL 5:
  To start your license server use the smartest_license script:
  
  /etc/init.d/smartest_license start

Results

You upgraded the FLEXlm binary executable files lmgrd, lmutil, and socbu to the latest version.

Replacing licenses after a server problem

When replacing licenses after a license server problem you must differentiate between two cases: Recovering and Rehosting.

Recovering licenses

Recovering a license file - as contrasted with "rehosting" a license file - is the process that enables you to reload your license file after you restore operation to a failed license server computer.

If the server crash is due to a hard drive failure, you should repair the hard drive, reinstall the license server software, restore a backup copy of the license file, and start lmgrd. (For a copy of lmgrd, see Licensing).
Other types of computer failures may or may not affect the integrity of the hostid. If the computer can be repaired without changing the hostid, reinstall the software and licenses, as needed, and start the license server. If the computer repair alters the hostid, then the licenses must be rehosted.
During the time required for repair of the license server, the emergency license capability can be used (see Emergency licenses).
CAUTION: Advantest highly recommends that you keep a backup copy of your license file in a safe location separate from your license server and test computer, so you can more easily recover a license should the need arise.

Rehosting licenses

Rehosting a license file is the process where all licenses you have originally locked to one license server are reissued by Advantest so a different computer can be used as a replacement license server.

Only two circumstances permit rehosting:

The catastrophic crash of your server.
Your server is inoperable due to obsolescence.

In either case you must contact your Advantest sales representative to initiate a rehost action to transfer your existing licenses to a new server. During the time this process takes you may need to utilize the emergency license capability (see Emergency licenses).

Note: Any device test already in progress when the server or network crashes, will continue to run for 2 hours. You will be unable to load new device tests.

To rehost, notify your Advantest sales representative that a catastrophic server failure has occurred or that you can no longer use the server due to obsolescence.

You must provide the representative with the hostid and hostname associated with the failed server, as well as the hostid and hostname of the new computer that you will use as your replacement license server.

You will receive a license file for the new license server computer that contains all the licenses associated with the failed license server. You will need to load the license server software on this computer.

Note: This process may take several business days, during which you can use an emergency license to keep your operation running (see Emergency licenses).

Emergency licenses

Emergency 10-day licenses enable you to meet test needs until your license is rehosted,

Note: Emergency 10-day licenses are only available for emergency circumstances.

The emergency license delivery mechanism will provide you with licenses immediately upon application via Advantest website.

Emergency licenses may be required:

If your server suffers a catastrophic crash that will require replacement of the server.
In the case of a server that has been catastrophically damaged, you will also have to rehost your original licenses. See Replacing licenses after a server problem. Rehosting can take several days during which time emergency licenses will meet your testing needs.

There are several considerations you should be aware of, if you apply for emergency licenses:

You must notify your Advantest sales representative as soon as possible by voice or email before you apply for a license.
You must take all reasonable steps to restore operation to your original license server, or replace it, within 10 days.
You should understand that temporary licenses you receive will be good for 10 days, by which time you will need to have rehosted your licenses.

See Licensing for further information about the emergency license process. You will need your STE support site login to access the support web page.

Adding licenses with lmreread

Adding licenses may be required for example, by a new device test, which uses extended test system features, or if you received emergency licenses.

About this task

You can add licenses to your license server without impacting device tests that are currently running.

To extend test system capabilities prior to running a new device test, follow the steps below.

Procedure

Install the additional licenses file on the server and networked computers on which they will run. See the "Floating license instructions" section of the "Site Planning and Preparation Guide" for the "V93000 Pin Scale System".
Use the lmreread command, which will initiate a reread of the license files on the server, which makes these added licenses available from the server's pool of licenses. See lmutil (lmreread) for more detail.
Shut down SmarTest. This will check the current licenses back into the server.
Restart SmarTest.

Results

SmarTest notices the new licenses that have been added to the server, and they will be available for device test.

You will need to shut down and restart SmarTest on each test system before that test system will be able to use the new licenses.

For complete lmreread syntax information see chapter 7 of the FLEXlm End User Guide. You can find the guide at the Advantest Web Site: Advantest Home > Global Services > Product Support > V93000 > V93000 Licensing > FLEXIm License Server Software Downloads and Documents > FLEXlm End Users Guide.