FNE / FNX troubleshooting

Where are the log files?

You can find the log files in the following folders for

How to update FNX/FNE license servers' software?

See also - Failover license server installation, upgrades and updates.

It is the advantage of the FNE/FNX server, that an update works without downtime. Do the following steps:

Note: If there is not a failover or tertiary server existing, you can skip the steps accordingly.

  1. Test the integrity of the downloaded SW on the primary license server:

    sh install-fne-<date>-<version>.sh test
    If you get the message "Package integrity verified" you can proceed with the other steps.

  2. Get emergency license for standby emergency license server and install it - just to be prepared

  3. Primary server: Do a dry-run installation on the primary server and look for messages. Example for the dry-run command:
    sudo sh install-fne-<date>-<version>.sh dry-run

  4. Primary server: If there are not any problems reported with the dry-run, do the update on the primary server. Example for the update command (no option used):
    sudo sh install-fne-<date>-<version>.sh

  5. Check e.g. with fne-status, that the primary server is running.

  6. Wait 2 minutes.

  7. Failover server: Do the update also on the failover server. See command above.

  8. Check e.g. with fne-status, that the failover server is running.

  9. Wait 2 minutes.

  10. Check with fne-status, if the FNE failover and (if installed) FNP triad is up again.

  11. Tertiary server: Do the update also on the tertiary server.

  12. Check e.g. with fne-status, that all servers are running. 

If there are any problems, please check, if a switch to the standby emergency license server is reasonable and open a LIST ticket to report the problem.

FNE registration problems

Error "Connection to https://license.advantest.com/flexnet/deviceservices failed with exception"
Issue description:

The user tried command

register-fne-server <id>

with the appropriate <id> and got the following error message:


Failed to activate your server
\{
"key" : "glsErr.connectionFailed",
"message": "Connection to https://license.advantest.com/flexnet/deviceservices failed with exception: exception: error on POST request for \ \"https://license.advantest.com/fL
exnet/deviceservices\": Connection reset; nested exception is javax.net.ssl.SSLException: Connection reset,*
: [ "https://license.advantest.com/flexnet/deviceservices, I/0 error on POST request for \"https://license.advantest.com/flexnet/deviceservices\": Connec
tion reset; nested exception is javax.net.ssl.SSLException: Connection reset ]*

Background information:

The connection to license.advantest.com with port 443 was not really available, but for the FNE server software it looked like that a connection is possible, therefor it tried and failed.

Solution: Enter the line

0.0.0.0   license.advantest.com

to file /etc/hosts.

FNE log file: errors and messages

Installation problems - error "/tmp/: failed to map segment from shared object: Operation not permitted"

Creation date: 08-Jan-2020
Author: Heike Klein

Issue description: The FNE server has been installed but cannot be started.
Log file error in /var/opt/flexnetls/socbu/logs/flexnetls.log.<date> - highlighted in red:

00:02:49,408 advantest-fne-server start
00:02:55,431 Used Options: /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.40-1jpp.1.el7.x86_64/jre/bin/java -Dbase.dir=/var/opt/flexnetls/socbu -Djsse.enableSNIExtension=false -server -Xms2g -Xmx2g -XX:CompressedClassSpaceSize=64m -XX:MetaspaceSize=256m -XX:+UseG1GC -XX:NewRatio=3 -XX:MaxGCPauseMillis=75 -XX:G1HeapWastePercent=10 -XX:InitiatingHeapOccupancyPercent=75 -XX:+CMSScavengeBeforeRemark -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED -XX:+ScavengeBeforeFullGC -Djava.security.egd=file:/dev/./urandom -jar /opt/flexnetls/socbu/flexnetls.jar -server-alias thbnk01fnxspl1v.th-bnk01.nxp.com -producer-settings /opt/flexnetls/socbu/producer-settings.xml -port 7070 -https-client-configuration /var/opt/flexnetls/socbu/outgoing-certs.conf
00:02:55,438 Environment: Version 2019.07.1.0-19315 / 7.6GB mem / 3.1GB disk free / 4x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz / Red Hat Enterprise Linux Server release 7.6 (Maipo)
00:02:58,835 INFO main Starting FlexNet License Server 2019.07 (build 256892)
00:02:58,836 INFO main Copyright (C) 2013-2019 Flexera Software LLC.
00:02:58,836 INFO main All Rights Reserved.
00:02:58,836 INFO main Running as a service
00:03:02,575 ERROR main Native code loading error java.lang.UnsatisfiedLinkError: /tmp/llsnc-2019.07.1.0-256892-amd64-libconnector.so(/tmp/llsnc-2019.07.1.0-256892-amd64-libconnector.so: failed to map segment from shared object: Operation not permitted)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1453)
at java.lang.System.load(System.java:570)

Reason for error:
the servers tries to use/exceute a shared library located in the /tmp directory. This is not allowed on the appropriate system. Running executables from /tmp is forbidden by the noexec option in the mount flags: check the /etc/fstab entry which looks like

/dev/mapper/vg00-tmp /tmp ext4 defaults,nodev,nosuid,noexec 1 2

Solution:
Ask IT to remove the noexec option to /tmp in /etc/fstab and remount it with
mount -o remount,exec /tmp
on all affected FNE license servers.

server filter checkout rejection

In the FNE log file flexnetls.log you maybe find the following message:
server filter checkout rejection
This message is basically saying the license requested is not available, f.e. for a specific serial number.
This is not an error of the server, its just the way the server returns a request for a license that is not present.

TS storage corrupted - FNE server cannot be started - FNE log error "FATAL main IO Exception: null"

In the FNE log file flexnetls.log you find the following entries:

03:36:16,826 INFO main Starting FlexNet License Server 2019.07 (build 256892)

03:36:16,827 INFO main Copyright (C) 2013-2019 Flexera Software LLC.

03:36:16,827 INFO main All Rights Reserved.

03:36:16,827 INFO main Running as a service

03:36:21,736 FATAL main IO Exception: null [90028-196]

 Note: you can extract the log file errors with command: showlog-fne-server -2

The error indicates a corrupted TS storage. In order to repair it, do the following:

  1. cd /opt/flexnetls/socbu

  2. sudo ./reset-fne-server -force

  3. If there is a failover license server configured you need to run the following command (not as root!) on the primary license server ./fne-failover-setup repair primary <failover_host.domain>
    This will repair the primary license server without touching the secondary license server.

    Note: the repair option is available since June 2020. It is not available if an older FNE/FNX installation script has been used and the SW has not been updated since then. Please check with fne-failover-setup -help if the option is available.

  4. If the failover server is corrupt, then you need to run the following command (not as root!) on the primary license server (this is not a typo - you need to run it on the primary license server!): fne-failover-setup repair failover <failover_host.domain> 

    Note: The command fne-failover-setup uses ssh/scp and sudo. You may be asked for your password several times.

Note: you can check the help page of fne-failover-setup with: fne-failover-setup -help

What is /var/opt/flexnetls/socbu/flexnetls_licenses.mv.db and can it be deleted?

/var/opt/flexnetls/socbu/flexnetls_licenses.mv.db is the database of the FNE server and should not be deleted at all.

What is directory /var/opt/flexnetls/socbu/logs/ used for and can it be deleted?

/var/opt/flexnetls/socbu/logs/ contain the log files of the FNE server and is needed for analysis if something is going wrong. The logs directory is always around 12GB big: it is cleaned up to 10GB each day and on head is the up-to-date log file.