Extending Advanced Monitoring

SAP provides Insights dashboards for Datadog, Insights monitors for Datadog, and metrics that are valuable for general use cases whether you're in an operations and integration, or applications support team. The Insights dashboards are based on core best practices from subject-matter-experts and feedback from the SAP community. You can extend the Insights Advanced Monitoring capabilities to adapt them to match your specific use cases or requirements.

Users with the Insights Advanced Manager or Insights Advanced Editor role can do the following to customize the Insights dashboards and Insights monitors:

  • Create your own Insights dashboards and then copy widgets from an existing dashboard to add to yours. For more information, see the New Dashboard in the Datadog documentation.
  • Clone any of the Insights dashboards as a starting point and then add and remove widgets to suit your monitoring requirements. For more information, see Cloning Existing Insights Dashboards for Datadog for Reuse.
  • Create custom views, add custom or existing Insights monitors, add custom widgets, customize existing widgets, and create filters on Insights dashboards you create or clone. For more information, see the following sections:

    For example, you could clone all of the Insights dashboards and then add filters to the widgets on the dashboards to reflect the needs of your team.

  • Create custom monitors and clone existing Insights monitors to customize them. For more information, see Creating Custom Monitors.
  • Mute monitors whose default thresholds don't meet your needs, or to limit the amount of notifications you receive from specific monitor types, see Muting Insights Monitors for Datadog.
  • Schedule downtime for monitors. Scheduling downtimes allows you to turn off notifications from specific or all monitors for a specified time, once or on a recurring schedule. Scheduled downtimes are useful for managing notifications during maintenance, or upgrades. See Scheduling Downtime.

Cloning Existing Insights Dashboards for Datadog for Reuse

Your business may have its own unique requirements, but you might find that a small modification to one of the Insights dashboards for Datadog may help optimize its use. The Insights dashboards provided by SAP cannot be modified. If you need to modify a Insights dashboards to meet your organization's requirements, SAP recommends cloning one of the Insights dashboards and adapting it as required.

Before you clone Insights dashboards, SAP recommends becoming familiar with how to create and modify dashboards in Datadog. In addition to cloning, you can create your own dashboards and then add metrics and monitors to your dashboard. In either case, here are some suggested topics to help you get started with dashboards in the Datadog documentation:

After you clone a dashboard, you can do further customizations, including:

Creating Custom Monitors

SAP provides predefined Insights monitors for Datadog that you can use on your dashboards. You can create your own custom monitors or clone and customize existing monitors. You can base monitors you create on any metric or log that is already parsed as a monitor in your account. Cloning existing monitors accelerates the process.

SAP provides several sample template monitors that Insights dashboards can pick up. You can use the sample template monitors to create your own custom monitors. The sample template monitors have the solace_customizable_template tag to distinguish them from the other active monitors provided with Insights. You can find an example of cloning and customizing a monitor using a sample template monitor in Cloning and Customizing Template Monitors .

The template monitors contain instructions for cloning and configuring your own monitors. If you clone and configure other monitors, you must follow the guidelines listed below. Not following the guidelines can cause SAP to be unable to properly manage your event broker services and cause inconsistencies in the monitoring data reported in the dashboards.

  • You must have the Insights Advanced Editor or Insights Advanced Manager role to create, clone, and configure monitors.
  • You should clone only monitors that have been tagged with solace_insights.
  • Do not clone monitors tagged with solace_rmms. Incorrectly cloning solace_rmms tagged monitors can cause SAP to be unable to properly manage your event broker services, which may result in degradation of service. If you clone any monitors that clearly indicated to not be copied, it can result in you temporarily losing access to your Datadog account and all changes in your Datadog account being reset.
  • After you clone a monitor, remove the SAP-specific tags (solace-*), such as solace-insights and solace_customizable_template. Removing the SAP-specific tags prevents your cloned monitor from being modified or removed during upgrades or content refreshes.
  • If you want a custom monitor to be picked up in by the Insights dashboards, add the custom_insights tag to the monitor.
  • Consider the following if you clone an active monitor:
    • you get notifications from the original SAP-provided monitor as well as the cloned monitor that you created
    • if you add custom_insights to a cloned monitor, it can result in both monitors appearing in the Insights dashboards

For more information about monitors in general, see Manage Monitors in the Datadog documentation.

Cloning and Customizing Template Monitors

SAP provides several sample template monitors that you can clone and customize to create your own custom monitors that log metrics that are of importance to your organization. With the template monitors you can define the event or system logs to monitor, the alert conditions, who is notified when those conditions are met, and what message they receive.

The template monitors are provided as is to enable you to create your own monitors. SAP does not offer or provide support for monitors created using the custom template monitor.

The following procedure creates a new monitor that notifies you when a client has connectivity issues.

  1. In Datadog, on the Monitors page, select the List tab and enter template in the search field to find the sample template monitors.

    Image displaying an example of the sample template monitor

  2. Select a sample template you want to clone. In this example, select [Insights] Sample template for broker event log monitors.
  3. In the title bar at the top of the screen, select Clone.

    Screenshot showing an example as described by the surrounding text.

    The cloned monitor opens in edit mode indicated by the breadcrumb ending with Configuration in the top left corner. The name of the monitor is prepended with Cloning. You can now define your monitor's new parameters.

  4. In section 1, define the Monitor scope. Select Query and use the Define the search query fields to define the query. For example, enter @event:SYSTEM_CLIENT_CONNECT_FAIL as your search query.

  5. Next define the criteria that triggers the monitor using the fields below the search query field. Click in the field to the right of by, and enter service_name(service_name) and client_username(@client_username). You may need to click the add icon to add an extra field. This triggers a notification whenever this combination appears in the selected logs.

    Image displaying the Defint the search query section

  6. In section 2, set the conditions that cause an alert to trigger: Use the fields in this section to set the thresholds that trigger alerts and warnings. Note that the default values for both the Alert and Warning thresholds are set to unrealistically high values to prevent the monitor from triggering, preventing Datadog from sending you unnecessary alerts. You should define different thresholds for each which are meaningful for your organization. When defining your thresholds, SAP recommends using 1 as a starting point.

    Image displaying the options in the Set alert conditions section

  7. In section 3, Configure notifications & automations you must:

    • complete the notification email that is sent, and who it is sent to

    • remove the following tags in metadata section:

      • solace_insights
      • solace_customizable_templates
      • version:*.*.*

    • add the custom_insights tag in the metadata section to prevent your custom monitor from being affected by future updates

    You can optionally configure when renotifications are sent, additional metadata to include with the notification, including priority and tags, and configure aggregation.

    For more information, see Notifications in the Datadog documentation.

  8. (Optional) In section 4, click Edit Access if you want to restrict who can view and edit the monitor. For more information, see Monitor Permissions in the Datadog documentation.

  9. Click Create.

You have successfully configured a monitor that alerts you whenever a client has issues connecting to your services.

Muting Insights Monitors for Datadog

You can mute Insights monitors for Datadog. When you mute Insights monitors, they continue to trigger on alerts and warnings, without sending out notifications. You may want to mute Insights monitors so you can be notified at different thresholds than those set by default. You can clone one of the Insights monitors, set a different threshold, and then mute the original monitor, receiving notifications from your customized monitor only.

Muting Insights monitors does not affect the ability of SAP to monitor the operation of your services. However, practice caution when muting Insights monitors to avoid missing notifications that are relevant to your monitoring needs. SAP provides some proprietary monitors that may not be mutable without having the correct roles and permissions. You can mute these monitors by scheduling downtime using their monitor tags. See Muting Locked Insights Monitors for Datadog Using Downtime for more details.

  1. In Datadog, go to Monitors > Monitor List to access the list of monitors. You may need to toggle the False setting flag under the Active filter to the left of the table to see a full list of available downtimes.
  2. Select the monitor you want from the list.
  3. Click on Mute Mute button in the top title bar to open the Create a downtime to mute this monitor dialog.

    Image displaying the downtime dialog in Datadog

  4. Use the dialog box to set the duration, scope, and an optional comment for the downtime, and then click Create Downtime. For more information, see Monitor Status Page in the Datadog documentation.

Quickly Mute and Unmute Monitors

You can quickly mute and unmute Insights monitors for Datadog directly from the list of Monitors using the quick mute icon quick mute icon.

Image displaying the location of the Quick Mute button

Muting a monitor this way does so for an indefinite time period. You generally cannot mute proprietary [Insights] monitors this way, see Muting Locked Insights Monitors for Datadog Using Downtime for details.

Muting Locked Insights Monitors for Datadog Using Downtime

You may encounter some Insights monitors that appear to be locked to users with the Insights Advanced Manager or Insights Advanced Editor role. These locked Insights monitors are generally proprietary monitors provided by SAP.

You can see if a monitor is locked by selecting it from the monitor list, clicking View Configuration, and scrolling to the Define permissions and audit notifications section of the monitor. A locked monitor displays a statement similar to 'Restricted. Access is restricted to some.'

Image displaying an example of how a locked monitor appears in the Define permission and audit notifications section

If the monitor is locked, it can still be muted by scheduling a downtime using its tags, see Scheduling Downtime.

Scheduling Downtime

You can schedule downtime for Insights monitors in Datadog. Scheduling downtime allows you to mute Insights monitors, from a scheduled start time, for a specific duration . Scheduled downtimes are useful when you plan upgrades or other work that may trigger notifications from the various Insights monitors. Perhaps you have a datacenter that has a maintenance scheduled on a specific weekday every week, causing notifications to be sent whenever the maintenance occurs. You can schedule a downtime for that period, muting notifications for that time. Once scheduled downtimes have been created, you can later edit, clone, or cancel them. For more information, see Edit, Clone, or Cancel a Scheduled Downtime.

Scheduling downtime does not affect the ability of SAP to monitor your services.

  1. In Datadog, go to Monitors > Downtimes to access the list of currently scheduled downtimes.

    Image showing location of Schedule Downtime button on Manage Downtimes tab of Monitor screen in Datadog

  2. Click schedule downtime icon to open the Schedule Downtime window.
  3. In step 1, choose the monitor to silence. You can select to schedule downtime for monitors by their monitor name or monitor tags using the Example of the By Monitor Name, By Monitor Tags toggle button toggle. You must use monitor tags for SAP provided monitors:

    • Select By Monitor Name to show the Monitor field where you can select a monitor from a list, or enter the monitor name if you know it.

    • Select By Monitor Tags to show the Monitor Tag field where you can select tags from a list, or enter a tag name if you know it.

      For SAP provided monitors (those tagged with solace_insights), you must use the monitor specific tag to schedule downtime for the monitor. For example, to schedule downtime for the Message VPN - Message Spool - Storage monitor, you must use the vpn_resource_message_spool tag unique to that monitor.

      You can view the tags attached to a monitor at the top of any open monitor, above the monitor's name.

      Image displaying the location of TAGS for a selected monitor

    • You can use the Group Scope field to mute by scope, such as service_id:some_id or queue_name:some_queue_name.
    • You can see monitors scheduled for downtime in the panel on the right of the Schedule Downtime page.
  4. In step 2, configure the schedule for the downtime. Downtimes can be one time events, or reoccurring. You can also specify the day, start time, and end time for the downtime.
  5. In step 3, enter notification text for Datadog to send to notify people about the scheduled downtime when it occurs. You can enter email addresses using @handle notation or select people to notify using the menu in step 4.

Edit, Clone, or Cancel a Scheduled Downtime

Once you have created scheduled downtimes, you can edit, clone, or cancel them as required.

  1. In Datadog, go to Monitors > Downtimes to access the list of currently scheduled downtimes.
  2. Hover the cursor over the schedule you want to change. A menu appears from the right side of the monitor.

    Image displaying the location of the Edit, Clone, and Cancel menu on the Monitors list

  3. Click the icon related to the task you want to do:
    • edit icon - Edit - Opens the Edit Downtime screen. This screen is identical to the Schedule Downtime screen detailed above (Scheduling Downtime) but with the fields already populated. You can change the fields as needed then click Save. You are returned to the list of scheduled downtimes with your edited downtime open.
    • clone icon - Clone - Opens the Clone Downtime screen. This screen is identical to the Schedule Downtime screen detailed above (Scheduling Downtime) but with the fields already populated. You can change the fields as needed and click Clone. You are returned to the list of scheduled downtimes with your cloned downtime open.
    • Cancel downtime icon - Cancel - Opens a confirmation window. Click Confirm to remove the scheduled downtime from the list.

Creating Custom Views

The default view for a dashboard shows everything in a predefined time interval without any filters applied. You can create different views to give you different contexts for the information on your dashboard, which can be useful for troubleshooting or analyzing your estate.

When you apply data or time filters to a view, that filtering is not saved. If you want to save the view based on your filtering choices, perform these steps:

  1. After applying filters in the Filter by fields, click the Saved Views drop-down list.

  2. Select Save selections as view to open the Save View dialog.

  3. Enter a name and an optional description, then click Save.

    The saved view is now accessible from the Saved Views drop-down list.

For more information about managing custom views, see Saved Views in the Datadog documentation.

Creating Filters

In addition to the some of the predefined filters that are provided on the Insights dashboards, you can also create your own filters using a template variable to create additional filters. Template variables can give you different views of the data, helping you better understand different aspects of your estate.

To create a template variable, perform these steps:

  1. On the dashboard, click the Edit next to the existing Filter by fields.
  2. Click the add icon to open the Variable type dialog.
  3. You can add both Filter and Group by variable types:

    • Filter: Add a tag or attribute to filter dashboard queries and visualizations.

    • Group by: Add a tag or attribute to show a breakdown of groups in your data.

    4. Group by is only supported on select widgets.

  4. Enter a name for the variable in the Variable name field.
  5. Click Save.

For more information, see the Template Variables in the Datadog documentation.