Diagnostic Data
This paper contains information about the diagnostic data available in
WebSphere Application Server V6. It contains information on the location of the
data, how it is collected, and configuration options.
It includes information on:
JVM logs (SystemOut and SystemErr)
Tracing
Collector tool
First Failure Data Capture (FFDC)
Process (native) logs
Service log (activity.log)
Installation logs
IBM HTTP Server and plug-in logs and traces
System management logs
WebSphere Rapid Deployment (WRD) logs
Carla Sadtler
David Titzler
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
2 WebSphere Application Server V6: Diagnostic Data
JVM logs (SystemOut and SystemErr)
SystemOut and SystemErr logs are created for every WebSphere® Application
Server process (application server, cluster member, node agent, and
deployment manager). These logs are known as JVM™ logs. The System.Out
and System.Err streams for each JVM are redirected to the SystemOut and
SystemErr logs. WebSphere Application Server writes to these logs and your
applications can also write to them by using the print(), println(), and
printStackTrace() methods.
You can find the logs in the following directory:
To configure the properties of these logs from the administrative console, select
Troubleshooting → Logs and Trace. Select the process whose logs you want
to configure, and then clicking on JVM Logs.
Figure 1 Changing the log file rotation properties
From here, you can edit:
WebSphere Application Server V6: Diagnostic Data 3
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
The file name (and the directory) of the SystemOut and SystemErr logs.
The file formatting; but we recommend that you leave this at the default value
of Basic to make the logs easier to read.
Log file rotation
The SystemOut and SystemErr logs are self-managing. They will write to the
specified file until either the maximum file size or a certain time is reached.
When that happens, the current log file is renamed as the current file name
plus the current timestamp. Then a new SystemOut or SystemErr file is
created for further logging. The older log files are called historical log files.
For example, after this occurs you may have the following SystemOut files in
the
– SystemOut.log
– SystemOut_05.06.07_10.28.48.log
The SystemOut.log is the current log file and the
SystemOut_05.06.07_10.28.48.log is the historical log file.
Depending on your needs, you can choose to have the log files rotate (roll
over) when they reach a specified size, a certain time interval, or both. If you
choose a time, we recommend that you specify 24 hours as the Repeat Time.
You can also set the Start Time to specify the time at which the logs rotate. If
you specify a file size, we recommend that you increase the Maximum Size
above its default of 1 MB. You’ll want to coordinate the value of Maximum Size
with the Maximum Number of Historical Log Files based on the available
disk space on your system. With either method, make sure that the amount of
log data that is saved is enough so that the relevant log data will still be there
when you identify that a problem has occurred.
Maximum Number of Historical Log Files.
The value that is entered here is the number of historical log files that are
kept. If the value is reached and another historical log file needs to be
created, the oldest one is removed from your system.
Installed Application Output
These properties affect how print and println statements from your
applications are output. There are two options here.
– Show application print statements. This is checked by default. If you
uncheck it, application print and println statements will not be logged to
the SystemOut and SystemErr log files.
– Format print statements. This is also checked by default. You can
uncheck it if you don’t want your application print and println statements
to be formatted like WebSphere Application Server messages in the log
files.
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
4 WebSphere Application Server V6: Diagnostic Data
All of these properties can be changed for both the SystemOut and the
SystemErr logs. You can choose to use the same properties for both logs, which
we recommend, or use different properties for them.
You can view the SystemOut and SystemErr files on the file system with a text
editor or you can view them within the administrative console. It may be useful to
view them in the administrative console if you need to view them from a remote
system. To view them in the administrative console, select the Runtime tab and
then click the View button next to the SystemOut or SystemErr file name.
The entries in the output of the SystemOut.log will be in the following format:
[7/12/05 14:46:00:264 EDT] 0000001a ApplicationMg A WSVR0221I:
Application started: adminconsole
Each entry can be deciphered as follows:
Timestamp
In the example, the timestamp is [7/12/05 14:46:00:264 EDT].
The timestamp is formatted using the locale of the process where it is
formatted. It includes a fully qualified date (for example MM/DD/YY), 24-hour
time with millisecond precision, and a time zone.
Thread ID
In the example, the thread ID is 0000001a.
The thread ID is an eight-character hexadecimal value generated from the
hash code of the thread that issued the message.
Short name
In the example, the short name is ApplicationMg.
The short name is the abbreviated name of the component that issued the
message. This is typically the class name of a WebSphere Application Server
component, and would be some other identifier for applications.
Event type
In the example, the event type is A.
The event type is a one character field that indicates the type of the message.
The possible values are:
– F - fatal message
– E - error message
– W - warning message
– A - audit message
WebSphere Application Server V6: Diagnostic Data 5
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
– I - informational message
– C - configuration message
– D - detail message.
– O - message that was written directly to System.out by an application or
internal components
– R - message that was written directly to System.err by the user application
or internal components
– Z - a placeholder to indicate that the type was not recognized
Message Identifier
In the example, the message identifier is WSVR0221I.
The message identifier that is a string that is nine characters in length and is
in the form CCCC1234X. The first four characters (CCCC) indicate the
WebSphere Application Server component that issued the message. The
next four characters (1234) indicate the specific message that is being issued
by the component. The last character (X) indicates the severity of the
message. Its value will either be I (informational), W (warning), or E (error).
Descriptions of all WebSphere Application Server message identifiers can be
found in the WebSphere Information Center item Troubleshooter reference:
Messages at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.
ibm.websphere.express.doc/info/exp/ae/welc_ref_trb_msg.html
Message
In the example, the message is Application started: adminconsole.
The message is the data that is logged to the SystemOut.log by the
component. It is meant to provide useful output for informational purposes,
debugging, and troubleshooting.
Tracing
Trace logs can also be configured in a manner similar to the JVM logs. Traces
must be explicitly enabled; they are disabled by default. Trace output gives very
detailed information about the execution of WebSphere Application Server code.
It provides timestamps, details about which WebSphere methods were called,
and special diagnostic data that is included to make troubleshooting easier.
To configure the properties of the traces from the administrative console, select
Troubleshooting → Logs and Trace. Select the process whose trace logs you
want to configure, and then click on Diagnostic Trace.
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
6 WebSphere Application Server V6: Diagnostic Data
Figure 2 Changing the trace properties
The configuration of the trace file properties is very similar to the configuration of
the JVM logs. You will want to sure that Enable Log remains checked. For Trace
Output, we recommend that you always select File instead of Memory Buffer so
that the trace logs will be easier to manage.
Trace files cannot be rolled over based on time. You must specify a Maximum
File Size in conjunction with the Maximum Number of Historical Files. You
should set these values appropriately depending on how long it may take to
reproduce the problem with trace enabled and how much disk space is available.
With these properties set, the trace files will roll over in the same manner as the
JVM logs. You can also specify a File Name and specify a directory for the trace
files.
As with the JVM logs, we strongly recommend selecting Basic (the default value)
for the Trace Output Format. This makes the trace easier to read and it is the
preferred format of the WebSphere Application Server Support team.
When you view the trace properties in the administrative console, you will notice
that there are two tabs, a Configuration tab and a Runtime tab. Tracing can be
enabled on either tab. The difference is that when you use the Configuration tab,
WebSphere Application Server V6: Diagnostic Data 7
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
you must restart the WebSphere Application Server process before the tracing
will begin. When you use the Runtime tab, tracing will begin as soon as you click
the OK or Apply button. In many production environments, it is preferable to
enable trace using the Runtime tab so that the WebSphere process does not
have to be restarted.
Once the trace properties are configured, you must decide which WebSphere
Application Server components to trace. To do this in the administrative console,
select Troubleshooting →Logs and Trace, select the process whose trace logs
you want to configure, and then click on Change Log Level Details. As with the
trace properties, the log level details can be set on the Configuration tab or the
Runtime tab.
Figure 3 Changing the log detail level in the administrative console
Here you can select which components and groups to trace. Components can be
any WebSphere Application Server packages or classes. Groups are pre-defined
sets of packages and classes that are useful for troubleshooting a particular
component. The default log detail level is *=info. The log detail levels in
WebSphere Application Server V6 are configured differently than the trace
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
8 WebSphere Application Server V6: Diagnostic Data
specifications in Version 5 and 5.1. If you use a Version 5 style trace
specification as the log detail level in Version 6, it will be mapped to the most
similar Version 6 log detail level. However, to ensure that the correct tracing is
enabled, we recommend only using the Version 6 log detail levels.
You can get a complete overview of the Version 6 log detail levels and how they
relate to the Version 5 and 5.1 trace specifications in the Log level settings
section of the WebSphere Information Center:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.
websphere.base.doc/info/aes/ae/utrb_loglevel.html
The MustGather documents for WebSphere Application Server components
discuss which specific log detail level to set for different types of problems. It is a
good idea to record the log detail level for different types of problems in your
diagnostic data collection plan. When setting the log detail level, you should set
the level to all in almost all cases. You should also include *=info in the
beginning of the log detail level so that informational logging is enabled for
components that are not being traced. For example, in order to enable a trace for
the J2C connection manager component, the log detail level would be set to:
*=info:WAS.j2c=all
You can specify as many components and groups as you wish. You can view the
components and groups that can be traced in the administrative console.
However, the more components and groups you trace, the larger the trace output
will be. It is a good idea to decide on a specific log detail level before enabling a
trace. The chapters of this redbook will help you determine which to use.
After enabling the trace, clear out any old trace files from the
configured the trace properties and log detail level properties on the
Configuration tab, you must restart the application server or process that you
want to trace. If you configured the trace properties and log detail level on the
Runtime tab, the tracing will start immediately without restarting the process. At
this point, you can reproduce the problem, and then disable tracing after you’ve
reproduced it. This will help ensure that the trace file does not grow too large and
it will make it easier to find the time at which the problem occurs in the trace.
In the trace output, you will see all of the messages that are also written to the
SystemOut.log, and you will also see trace events. The trace events will be in the
following format:
[7/12/05 16:13:10:379 EDT] 00000032 DSConfigurati > getPooledConnection Entry
Each entry can be deciphered as follows:
Timestamp
WebSphere Application Server V6: Diagnostic Data 9
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
In the example, the timestamp is [7/12/05 16:13:10:379 EDT].
The timestamp is formatted using the locale of the process where it is
formatted. It includes a fully qualified date (for example MM/DD/YY), 24-hour
time with millisecond precision, and a time zone.
Thread ID
In the example, the thread ID is 00000032.
The thread ID is an eight-character hexadecimal value generated from the
hash code of the thread that issued the trace event.
Short name
In the example, the short name is DSConfigurati.
The short name is the abbreviated name of the component that issued the
trace event. This is typically the class name of a WebSphere Application
Server component, and would be some other identifier for applications.
Event type
In the example, the event type is >.
The event type is a one character field that indicates the type of the trace
event. The possible values are:
– > - indicates the entry of the specified method name
– < - indicates the exit of the specified method name
– 1 - a trace entry of type fine or event
– 2 - a trace entry of type finer
– 3 - a trace entry of type finest, debug, or dump
– Z - a placeholder to indicate that the trace type was not recognized
The example indicates that the getPooledConnection method is entered.
Class name
The class name is an optional part of the trace entry. It indicates the class that
generated the trace event. In the example, the class name does not appear.
Method name
The method name is another optional part of the trace entry. It indicates the
method that generated the trace event. In the example, the method name is
getPooledConnection.
Text message
In the example, the message is Entry.
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
10 WebSphere Application Server V6: Diagnostic Data
The text message is the data that is written to the trace output file. It is meant to
provide advanced debugging and troubleshooting information.
Parameters
Optionally, parameters can also be included in the trace entry. In the
example, there are no parameters.
Collector tool
The WebSphere Application Server collector tool is a script that can be found in
the
the script produces a Java™ archive (jar) file that contains all of the logs and
XML configuration files from your WebSphere Application Server installation, as
well as operating system and Java information, and data on whether or not the
software prerequisites were met and their levels.
The collector tool should always be run under the root or administrator user ID
and it must be run from a directory outside of the
The resulting jar file will be created in the current directory and the file name will
be
C:\tmp\ServerHost1-Cell01-CellManager01-Dmgr01-WASenv.jar
The resulting jar file is very useful to the WebSphere Application Server Support
team and any other people who are involved in the problem determination
process. It allows them to quickly view your WebSphere Application Server
configuration and see any errors or exceptions that have occurred.
More information about the collector tool can be found in the WebSphere
Information Center items:
Gathering information with the Collector tool at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.
ibm.websphere.base.doc/info/aes/ae/ctrb_ct.html
Running the collector tool
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.
ibm.websphere.base.doc/info/aes/ae/ttrb_runct.html
Analyzing collector tool output
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.
ibm.websphere.base.doc/info/aes/ae/ctrb_readct.html
WebSphere Application Server V6: Diagnostic Data 11
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
First Failure Data Capture (FFDC)
WebSphere Application Server V6 includes a feature called First Failure Data
Capture (FFDC). The FFDC feature runs in the background and collects events
and errors that occur during WebSphere Application Server runtime. The
information that it collects are written to log files in the
FFDC does not affect the performance of WebSphere Application Server and
should not be disabled. The FFDC logs will most likely not be useful in your
problem determination efforts, but they may be useful to the WebSphere
Application Server Support team if you open a PMR.
There are three FFDC configuration files in the
directory. The only file that you should modify is the ffdcRun.properties file. You
can add the property ExceptionFileMaximumAge to the file. This property specifies
the number of days that an FFDC log will remain in the
As part of your diagnostic data collection plan, you may want to modify the
ExceptionFileMaximumAge to ensure that the FFDC files remain on your system
for a certain time period. You should not modify any other properties unless you
are asked to do so by the WebSphere Application Server Support team.
More information about the FFDC feature can be found in the WebSphere
Information Center item First failure data capture at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.
websphere.base.doc/info/aes/ae/ctrb_ffdc.html
Other logs
The following logs will not be useful to all WebSphere Application Server,
Version 6 problem determination efforts, but they are useful for some types of
problems as described in the chapters of this redbook.
Process (native) logs
Native code running in a WebSphere Application Server process can write data
to the process logs (also called native logs). Native code is non-Java code,
typically found in files with .dll, .exe, and .so extensions. The process logs are
named native_stdout.log and native_stderr.log. They are located in the
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
12 WebSphere Application Server V6: Diagnostic Data
The only configuration that is possible for the process logs is changing the
directory location or file names for the logs. You can do this in the administrative
console. Select Troubleshooting →Logs and Trace, select the WebSphere
Application Server process, and then select Process Logs.
More information about the process logs can be found in the WebSphere
Information Center item Process logs at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.
websphere.base.doc/info/aes/ae/ctrb_stdlogs.html
Service log (activity.log)
The service log is more commonly known as the activity.log and is found in the
activity.log for each node. WebSphere Application Server runtime events are
logged to the activity.log. It is written in binary format, so it cannot be viewed in a
text editor. The main purpose of the activity.log is that it can be viewed with the
Log Analyzer tool, is a graphical user interface that displays the events from the
activity.log and uses a symptom database to analyze the events and diagnose
problems.
More information about the Log Analyzer can be found in the WebSphere
Information Center item Log Analyzer at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.
websphere.base.doc/info/aes/ae/ctrb_jfla.html
It is also possible to view the events in the activity.log outside of the Log Analyzer
by using the showlog script in the
Details on the usage of the showlog script can be found in the WebSphere
Information Center item Showlog Script at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.
websphere.base.doc/info/aes/ae/rtrb_showlog.html
You can configure properties of the activity.log in the administrative console.
Select Troubleshooting →Logs and Trace, select the WebSphere Application
Server process, and then select IBM® Service Logs. You can select whether to
enable or disable the activity.log, choose the directory location and file name, set
the maximum file size, and select what types of messages will be logged.
More information about the service log can be found in the WebSphere
Information Center item Viewing the service log at:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.
websphere.base.doc/info/aes/ae/rtrb_viewsvclog.html
WebSphere Application Server V6: Diagnostic Data 13
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
Installation logs
When you have a problem installing WebSphere Application Server, Version 6,
you may need to view the following logs to determine the failure causes.
This log file records the installation status
This log file records the profile creation status
This log file records the events of install verification test.
IBM HTTP Server and plug-in logs and traces
When you have a problem relating to the IBM HTTP Server or the Web server
plug-in, you may need to view the logs or enable a trace. Here are the details
about these logs and traces.
IBM HTTP Server logs
The IBM HTTP Server writes two log files, an access log that contains details of
all accesses to the Web server and an error log that contains details of any
errors. The default location of the logs is as follows:
Windows®
– Access log:
– Error log:
UNIX®
– Access log:
– Error log:
Web server plug-in logs
The plug-in also writes its own log which can be found in the Web server plug-in
install directory path. The log you are looking for will be under another directory
structure named for the logical Web server as defined in the WebSphere
configuration.
You can find the location of the log file by first looking at the Web server
configuration. This refers to the plug-in configuration file as shown in Example 1
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
14 WebSphere Application Server V6: Diagnostic Data
on page 14. The plug-in configuration file then tells you where the log file is as
shown in Example 1. This example also shows you where you set the amount of
detail that is logged.
Example 1 Location of plug-in log file
The default setting for LogLevel is Error, but you can set it to Trace to collect
significantly more information. Should you need to raise this problem with IBM
Support, they will request a plug-in trace.
Web server plug-in trace
To get an effective trace, you will need to enable as much logging as possible on
the Web server. For example, you can set the logging level to debug to capture
verbose output in the IBM HTTP Server by modifying the LogLevel directive in
the configuration file as shown:
LogLevel debug
You will need to restart the IBM HTTP Server for this to take effect.
Enable trace logging in the Web server plug-in by setting the LogLevel directive
in the plugin-cfg.xml file as shown:
You do not need to restart the IBM HTTP Server for this to take effect.
Network trace
In rare cases, you might need to use a network protocol analyzer that will allow
you to capture an iptrace. This may help you to determine where the problem
lies. WebSphere Application Server does not supply such a tool, but there are
third party tools available (for example, Ethereal from
http://www.ethereal.com/).
Tip: The plug-in trace generates significant amounts of data. Make your test
as specific as possible and run it in isolation to reduce the number of lines
generated.
WebSphere Application Server V6: Diagnostic Data 15
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
System management logs
When you have a system management problem, you may need to view certain
logs or enable a trace. Here are the details about these logs and traces.
Output from wsadmin
Messages from wsadmin are written to the wsadmin.traceout log file:
You can also increase the amount of data logged to this file by tracing the
wsadmin utility. Update the following file:
Uncomment the line:
com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled
Note that the information logged will be of limited use as wsadmin calls MBeans in
the application server running the administrative console application, so you will
usually need to trace that application server as well.
Management scripts
WebSphere Application Server services can be managed using the supplied
management scripts. For example, each WebSphere Application Server
installation has a script to start an application server, a script to stop an
application server and a script to show you the status of all application servers
defined in a profile. Each of these scripts writes its own log file into the server’s
logs directory. For example, the stopServer script writes stopServer.log into the
logs directory:
Profile management logs
The profile creation and management tool wasprofile writes messages to the
profile independent logs directory, that is:
This log file is in XML format.
The Java graphical interface that is used to create a profile simply calls the
wasprofile command after collecting the information needed. By default it does
not write a log, but you can pass it a log parameter as shown:
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
16 WebSphere Application Server V6: Diagnostic Data
pctWindows -is:log c:\temp\pct.log
WebSphere Rapid Deployment (WRD) logs
The WRD tool works on a directory that you create and pass to WRD in an
environment variable WORKSPACE. It logs Eclipse messages into two separate files
within this directory:
Like other WebSphere Application Server utilities, WRD calls MBeans on the
application server. The application server logs can help you resolve a problem
with WRD. There is no way to trace the WRD utility but you can trace the
application server as described in “Tracing” on page 5.
Summary of logs
Table 1 shows a summary of the WebSphere logs.
Note: In fhe following table:
Application Server, for example :
c:\Websphere\Appserver, etc.
Application Server profile, for example:
for example:
c:\IBM HTTP Server
WebSphere Application Server V6: Diagnostic Data 17
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
Table 1 Log file summary
Tasks Logs Format / tools
Installation tasks
WebSphere Installation
IBM HTTP Server installation
ihsv6_install.log
gskitInstall.log
text
Sample and IVT application
installation:
DefaultApplication
ivtApp
query
PlantsByWebSphere
SamplesGallery
defaultapp_config.log
defaultapp_deploy.log
ivt_config.log
query_config.log
samples_config.log
samples_install.log
text
WebSphere system application
installation:
filetransfer
filetransferSecured
ManagementEJB
SchedulerCalenders
adminconsole
filetransfer_config.log
mejb.log
scheduler.cal_config.log
webui_config.log
text
Profile tasks
Profile creation wizard
file/wasprofile_create_
og.
XML
Application and system tasks
Application print() and println()
SystemOut.log1
SystemErr.log1
Text.
administrative console:
Troubleshooting → Logs and
Trace →
JVM System.out and System.err
streams
1 Configurable
2
deployment manager. If the profile is for a node agent,
deployment manager,
4085paper.fm Draft Document for Review August 29, 2005 3:15 pm
18 WebSphere Application Server V6: Diagnostic Data
WebSphere processes Process logs at
native_stderr.log1
native_stdout.log1
Text (the server must be stopped
to view with a text editor)
administrative console:
Troubleshooting → Logs and
Trace →
Logs
WebSphere System.out stream
+ messages that contain
extended service information
IBM service log (aka activity log)
at
Binary format.
View with Log Analyzer. Showlog
tool can convert the contents to a
text format.
Operational tasks
Start / stop an application server.
SystemOut.log1
SystemErr.log1
If using startServer
-trace, see
startServer.log
stopServer.log
text
Start / stop a deployment
manager
SystemOut.log1
SystemErr.log1
startServer.log
stopServer.log
text
Start / stop a node agent
:
SystemOut.log1
SystemErr.log1
If using startNode -trace, see
:
startServer.log
stopServer.log
text
Tasks Logs Format / tools
1 Configurable
2
deployment manager. If the profile is for a node agent,
deployment manager,
WebSphere Application Server V6: Diagnostic Data 19
Draft Document for Review August 29, 2005 3:15 pm 4085paper.fm
Start / stop a cluster When you start or stop a cluster,
that action is taken on each
server. The logging is the same
as if you started or stopped each
server.
Configuration tasks
Adding a node to a cell
addNode.log
runAddNode.log
text
Tasks Logs Format / tools
1 Configurable
2
deployment manager. If the profile is for a node agent,
deployment manager,
No comments:
Post a Comment