Steve Spencer's Blog

The blog of Steve Spencer

Windows Azure Diagnostics

Diagnostics in any application is a necessity and Windows Azure applications are no different. You could remote desktop onto the instance and check the event logs and even run up debug view so that you can see your system diagnostic messages. There is however a mechanism provided to retrieve a whole load of diagnostic information. By enabling Windows Azure Diagnostics you can retrieve the diagnostic trace logs, windows event logs, performance counters and other useful data. Enabling Windows Azure Diagnostics can be done by setting the Enable Diagnostics check in the configuration of each Azure role. You also need to add a diagnostic connection string as follows:

image

This enables diagnostics but this alone will not provide you with any information. Windows Azure diagnostics works by capturing the data that you have requested and periodically transferring it to table or blob storage where you can view the information. The information can be requested by configuring the diagnostics you require either in code or in a config file. The config file resides in blob storage and can be changed at runtime so it make sense to use that configuration mechanism rather than code as it can be easily turned off when not in use.

Firstly you may want to retrieve your tracing information that is traced using System.Diagnostic.Trace. the easiest way to do this is to add a trace listener. The Windows Azure SDK contains one that can be used. this is called DiagnosticMonitorTraceListener. This can be added to the web.config file in the same way as other listeners or via code. When I added it to the web config file I had issue on certain projects where the Windows Azure Diagnostics assembly could not be found. Adding the configuration in code always seemed to work. As you will need to redeploy anyway to update your web config file in Windows Azure it makes little difference whether the configuration is in code or the web config file (except that you need to rebuild). In order to add it to code I added the following line to the  global.asx.cs file:

   1: void Application_Start(object sender, EventArgs e)
   2: {
   3:     // Code that runs on application startup
   4:     System.Diagnostics.Trace.Listeners.Add(new Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener());
   5: }

You now need to modify the config file located in blob storage. The config file is located in blob storage inside the wad-control-container. There will be a config file in here for each deployment you have made with diagnostics enabled and for each instance. You will need to update each instance you wish to receive diagnostic information from.

Tracing can be enabled by modifying the following xml accordingly:

   1: <Logs>
   2:   <BufferQuotaInMB>1024</BufferQuotaInMB>
   3:   <ScheduledTransferPeriodInMinutes>5</ScheduledTransferPeriodInMinutes>
   4:   <ScheduledTransferLogLevelFilter>Verbose</ScheduledTransferLogLevelFilter>
   5: </Logs>

This will transfer the logs at 5 minute intervals at the highest log level. You can change the level to be Information or Error which represents whether you have used TraceInformation or TraceError whereas Verbose traces both and using Trace.WriteLine. The logs will be transferred to table storage in a table called “WADLogs”

Performance counters can be configured as follows:

   1: <PerformanceCounters>
   2:   <BufferQuotaInMB>0</BufferQuotaInMB>
   3:   <ScheduledTransferPeriodInMinutes>5</ScheduledTransferPeriodInMinutes>
   4:   <Subscriptions>
   5:     <PerformanceCounterConfiguration>
   6:       <CounterSpecifier>\Processor(_Total)\% Processor Time</CounterSpecifier>
   7:       <SampleRateInSeconds>30</SampleRateInSeconds>
   8:     </PerformanceCounterConfiguration>
   9:     <PerformanceCounterConfiguration>
  10:       <CounterSpecifier>\Memory\Available Bytes</CounterSpecifier>
  11:       <SampleRateInSeconds>30</SampleRateInSeconds>
  12:     </PerformanceCounterConfiguration>
  13:   </Subscriptions>
  14: </PerformanceCounters>

This will transfer the processor percentage and available bytes every 5 minutes with a 30 second sampling frequency. These are stored in table storage in a table called “WADPerformanceCountersTable”. Information about the performance counters that are available can be found here:

Event logs can be configured as follows:

   1: <WindowsEventLog bufferQuotaInMB="0"
   2:      scheduledTransferLogLevelFilter="Verbose"
   3:      scheduledTransferPeriod="1">
   4: <!-- The event log name is in the same format as the imperative 
   5:        diagnostics configuration API -->
   6:     <DataSource name="Application!*" />
   7:     <DataSource name="System!*" />
   8: </WindowsEventLog>

These are stored in a table called “WADWindowsEventLogsTable”.

Further information can be found at:

Example configuration file

Overview of Storing and Viewing Diagnostic Data in Windows Azure Storage

Windows Azure Roles Fail to run when deployed to Azure

Recently I was helping out at the Azure Bootcamp in London and during the labs a common theme kept occurring when the labs were deployed to a real Azure account. The roles failed to run and it appeared that the deployment was taking forever.  This is something I experienced first hand when I was starting out with Azure. There is a way to diagnosing these deployment errors and it is by using IntelliTrace. During deployment you can enable IntelliTrace as part of the publish dialog

image

The IntelliTrace option is only available if you have Visual Studio 2010 Ultimate. Once deployed to Azure the Roles will attempt to start and any errors during this phase will lead to the symptoms mentioned above. You can then connect to your Azure environment using the Server Explorer in Visual Studio to retrieve the IntelliTrace files which can be opened in Visual Studio and show any exceptions that may have been thrown. Further information can be found here. Once you have diagnosed your issue please ensure at you then disable the IntelliTrace by redeploying the fixed application as it will have a negative impact on performance if left enabled.

 

Getting back to the problem we have at the Bootcamp, the issues was that the deployed application was trying to writing information to Azure storage and the connection string was still pointing to Development storage. This was strange because none of the deployed applications had got to the Azure storage part of the lab so you would have thought that there was no need for a connection string. Luckily I had the exact same problem with one of my earlier deployments and it turns out that when a project is created the Diagnostic plug-in is automatically enabled. The diagnostic plug-in requires its own connection string to Azure storage so that the diagnostic information can be stored. Looking at the role configuration in Visual Studio you can see the Diagnostic plug-in configuration.

image

To fix the deployment issue click the button next to the connection string text box and enter the details of your Azure Storage account.

image

You will need to redeploy the application or upload the new ServiceConfiguration.cscfg to fix this issue. If this still does not resolve the issue then try disabling the Diagnostics plug-in and redeploy.

Logging with Log4Net

A while ago I was looking for an alternative to System.Diagnostics.Trace and found Log4Net. I know the patterns and practices group have created enterprise logging but I find Log4Net nice and easy to use. I created some documentation and samples on my website at http://www.recneps.co.uk/log4net.aspx.

The BlackMarble.Diagnostics.Logging libraries encapulate both System.Diagnostic.Trace and Log4Net so that we can use a standard way of logging and then configure at runtime one we want to use.