Logging and debug flag options for Access Server

Where log information can be found for debugging

There are log files on the client, which are most useful for figuring out why a client is having problems making a connection to a server, and figuring out which routes and instructions it is receiving. And there is the Log Reports section in the Admin UI which is generally used to figure out when a user connected, for how long, when people logged onto the web interface, how much data they've used, and if there were any simple type of errors when authenticating and connecting. On the server there are log files that contain technical information, and this technical information can also instead be sent to syslog locally. If you want it sent to a remote server, configure a rule in the local syslog daemon to redirect it to a networked syslog server. The technical information contained in the server logs can be expanded to include various extra information. To do this, for specific functions in the Access Server, there are debug flags, which can be activated in as.conf. This is explained on this page as well, further down.

Locating the client log files

There are log files on the client, which are most useful for figuring out why a client is having problems making a connection to a server, and figuring out which routes and instructions it is receiving.

Log file location for the OpenVPN Connect Client for Windows:
C:\Program Files (x86)\OpenVPN Technologies\OpenVPN Client\etc\log\openvpn_(unique_name).log

The OpenVPN Connect Client for Mac:
/Library/Application Support/OpenVPN/log/openvpn_(unique_name).log

Macintosh may not show you this folder in finder as it only shows you certain things and hides others. So to get to the /Library folder, open Finder and in the menu at the top choose Go followed by Go to folder and then enter the path /Library to get into that directory. You can then go to the correct folder and look up the log file. Please also note that the OpenVPN Connect Client for Macintosh will have permissions set on the log file so that you cannot normally open it. To bypass this, right click the log file and choose the Get info option in the menu. Then at the bottom, under Sharing & Permissions, you will be able to use the yellow padlock icon to unlock the settings and to give everyone read access. Then you will be able to open the log file with a right click and selecting Open with and then choosing something like Text editor to view the contents of the log file.

Locating the server log files

On the server there are log files that contain technical information. The technical information contained in the server logs can be expanded to include various extra information. To do this, for specific functions in the Access Server, there are debug flags, which can be activated in as.conf. This is explained on this page as well, further down.

On the OpenVPN Access Server there is the server side log:
/var/log/openvpnas.log
/var/log/openvpnas.node.log 
(in case of a failover setup)

In the event that you are having problems with starting the Access Server or certain portions of it, for example the web services, then it may be useful to stop the Access Server service, move the log file aside, then start the Access Server service, and stop it again immediately. This creates a new clean log file that contains the startup and shutdown sequence of the Access Server and no other extraneous information. This makes analysis of the log file much easier. To do so use these commands in order:

service openvpnas stop
mv /var/log/openvpnas.log /var/log/openvpnas.log.old
service openvpnas start
service openvpnas stop

You can then grab the /var/log/openvpnas.log file for analysis and start the Access Server again:

service openvpnas start

Setting up log rotation for /var/log/openvpnas.log.*

OpenVPN Access Server normally keeps on logging until the disk is full. What you can do instead is to set a threshold for the log file to be rotated out, meaning a threshold in bytes after which openvpnas.log gets renamed to openvpnas.log.1, and a new openvpnas.log file is started for logging purposes. Basically it creates a new log file and the old one gets renamed to .1, .2, .3, etc, as time goes on. It's always sequential meaning that .1 is more recent than .2. The log file with the highest number will be the oldest file - it just keeps bumping up the file names.

Once the threshold is set, you can then set up a cron job that runs every so often and clears out any really old files. The amount of files you choose to retain times the file size of the log rotation setting determines how much log data you're going to be retaining in total, ensuring you never go over a certain amount of bytes used for the OpenVPN Access Server's log files.

Note: you can also simply log to syslog which is explained below, and syslog should always already have rotation rules set on it in the operating system, that cleans it up regularly.

We assume that you are logged on to the server with root privileges when doing any command line tasks on the Access Server.

Open as.conf for editing in nano text editor:

nano /usr/local/openvpn_as/etc/as.conf

At the bottom add this line. The number represents the amount of bytes. Default is 1048576 (1 megabyte):

LOG_ROTATE_LENGTH=1048576

Press ctrl+x, then press y, and then press enter, to save and exit the file. Then restart the Access Server service:

service openvpnas restart

Now your Access Server will make log files of the specified file size. But that doesn't clean up old log files yet. To do that, set up a cronjob rule that, for example, once a day clears out any log files that are numbered .15 or higher at 4 am every night. Adjust the command as you like to set your own limits and time of execution.

Open the crontab file for the account you are logged on as:

crontab -e

When doing this for the first time you may be asked which text editor to use. We tend to advise nano as it's easy to use. At the bottom of the crontab file add this line:

0 4 * * * rm /var/log/openvpnas.log.{15..1000} >/dev/null 2>&1

Press ctrl+x, then press y, and then press enter, to save and exit file (if you use nano).

Now every night at 4 am, this script will delete files called /var/log/openvpnas.log.15, and the way through to /var/log/openvpnas.log.1000. That should pretty much guarantee that you only end up with the main log file and 14 extra older log files.

Logging to syslog instead of the standard log file

This is a configuration setting that enables logging to the local syslog daemon. Instead of logging to a file it logs to syslog instead. If you want to redirect to another syslog server on the network you can configure the operating system's syslog daemon to redirect any OpenVPN Access Server service syslog line to an external network syslog server. All syslog lines regarding Access Server will contain the keyword openvpnas in it so it is possible to filter for this with a rule in the syslog daemon, and forwarding only that information.

Open the as.conf file for editing:

nano /usr/local/openvpn_as/etc/as.conf

At the bottom add this line, making sure it's CAPITALIZED:

SYSLOG=1

Press ctrl+x, then press y, and then press enter, to save and exit the file. Then restart the Access Server service:

service openvpnas restart

It will now log to the syslog daemon, which by default is logging to the file /var/log/syslog.

Redirecting to an external syslog server

We're going to be assuming you're using the Ubuntu operating system, so these instructions are for that OS only. It's probably similar or the same on other operating systems too but you may have to look up documentation and make adjustments as needed. The appliances we provide are currently all based on Ubuntu so these instructions should work for you if you use one of our prepared systems on Amazon AWS, HyperV, or ESXi.

Create a file for the rsyslog daemon rule:

nano /etc/rsyslog.d/openvpnas.conf

The file should be a new empty file. Add this line to log to an external UDP syslog system:

if $programname == 'openvpnas' then @remote.syslog.server

Or instead this line if it is an external TCP syslog system:

if $programname == 'openvpnas' then @@remote.syslog.server

Press ctrl+x, then press y, and then press enter, to save and exit the file. Now restart the syslog daemon:

service rsyslog restart

As an aside, you can also instead of supplying a remote server address like @remote.syslog.server simply specify another file like /var/log/myownfilename.log and it will log it there instead.

A list of debugging flags

If you don't know what you're doing then the safest is to say: don't use these. Normally only OpenVPN Inc. support personnel are the ones that use these debugging flags, or to advise customers to use a particular debug flag, when there is a specific need to debug a particular problem. But we've decided to make some of the more useful debug flags available to the general public, because some can be useful in getting more data from the Access Server for purposes other than debugging, although it has its uses to solve problems as well of course.

Some of these debug flags can greatly increase the amount of logging data produced by the OpenVPN Access Server, so beware filling your hard drive with log data and running out of disk space. Not all flags produce a lot of information, but some do. And some of them even log password data or session data to the log, so beware of this. Therefore it's usually best to use some of these flags to pinpoint a problem, get log data, and then disable the debug flag.

Most debug flags are set in the /usr/local/openvpn_as/etc/as.conf file by adding it at the bottom of the file, and cold restarting the Access Server service afterwards with this command:

service openvpnas restart

DEBUG_AWSINFO=1

Logs extra information in liman info output and in /var/log/openvpnas.log regarding the licensing process when using an Amazon AWS prelicensed tiered instance. Especially in the case of problems reaching a license activation server, the output found here will be useful to determine what the issue is. See also the troubleshooting section for the AWS tiered instance licensing system.

LOG_N_CLIENTS_CHANGE=1

Whenever the internal currently connected users count is altered, the log system now mentions this alteration. This can be useful if you suspect the connected user count is off for whatever reason. An example line from the log file:

0000-00-00 00:00:00+0000 [-] ***** N_CLIENTS CHANGE 0 -> 1

FAVOR_LZO=1

This is a debug flag to override the order in which compression algorithms are chosen for connecting clients. Forces the use of LZO. In extremely rare cases can help to resolve connectivity problems from iOS devices with very specific compression problems.

API_TRACE_SA=1

If you ever have the suspicion somebody or something is making alterations to your configuration settings without your knowledge, like for example a colleague with access to the system, or a browser plugin that is supposed to only help you with some tasks like filling in forms or remembering passwords for you, but is instead messing with settings you didn't touch while you're working on other settings, then this debug flag is useful. Or if you just want to log all the changes to the configuration settings. This debug flag logs all the activity between Access Server and the configuration databases. An example line from the log file showing that the user openvpn is signing on to the admin UI successfully:

2017-09-19 17:11:54+0200 [-] *** API CALL f=authenticate args=[{'username': 'openvpn', 'password': '[redacted]', 'client_ip_addr': '12.34.56.78'}, {'log_service_name': 'WEB_ADMIN', 'request_superuser_privileges': True}] time=0.012

DEBUG_LOGDB=1

Normally the Log Reports page and its contents are stored in the log.db database file on the Access Server, and this is kept out of the log system. The reasoning here is that the log system is used for tracking and resolving problems with the OpenVPN Access Server program itself, not to keep track of who logs in and how much bandwidth they're using. That is what the log database, the Log Reports page, and the logdba command line tool are for. However if for some reason you want to log everything that goes into the log database to the log system as well then this is the flag to use. An example line from the log showing that the user openvpn has successfully logged on to the admin UI web service:

0000-00-00 00:00:00+0000 [-] LOG ERR: 'LOG_DB RECORD {"username": "openvpn", "node": "OPENVPNAS", "service": "WEB_ADMIN", "real_ip": "12.34.56.78", "timestamp": 1505833476, "start_time": 1505833476, "session_id": "u1OfDeOuagO1sGQg", "auth": 1}'