Tutorial: Discover patterns in data

Learn how to find patterns and trends in data from feature datasets using density, hot spot, and outlier analyses.

Parking enforcement incidents: hot spot, outlier, and density pattern analyses.

Analyze pattern analyses allow you to perform complex geometry, attribute, and statistic calculations to identify spatial patterns and relationships in feature data. All operations result in new feature data.

In this tutorial, you use the Calculate Density, Find Hot Spots, and Find Outliers operations in either Map Viewer or programmatically using the ArcGIS Python, ArcGIS REST JS, and ArcGIS REST APIs. The results identify areas where there are statistically higher occurrences of parking violations in San Francisco.

The analyses include:

Calculating the density of incidents per square mile.
Finding statistically significant hot and cold spots.
Finding statistically significant outliers in the data.

Prerequisites

Steps

Copy the web map

The tutorial web map contains predefined layers to use as the starting point for the analyses outlined in the steps below.

Go to the Analyze patterns in data tutorial web map and click Sign in.
Verify that you have the following layers by toggling the visibility on and off:
- Driveway and sidewalk parking violations
- Boundary
In the Content (dark) toolbar, click Save and open > Save as to save a copy in your portal.

Calculate density

There are 15,351 features in the Driveway and sidewalk parking violations hosted feature layer that represent reported parking violations throughout San Francisco. Before performing a statistical analysis, it is helpful to understand how the features in the feature layer are distributed. Use the Calculate Density operation to determine where there are parking incidents on a scale of least to most dense per square mile.

To learn more, go to Calculate density.

Steps to use the Map Viewer

Steps to use ArcGIS Python, REST JS, and REST APIs

In the Settings (light) toolbar, click Analysis > Tools > Calculate Density.
Set the following parameters:
- Input point or line features: Driveway and sidewalk parking violations.
- Expand Optional layers > Clip output to: Boundary
- Classify by: Natural breaks
- Output name: Calculate density of incidents.
Click Estimate credits. The estimated cost for ArcGIS Location Platform is USD $1.53 and ArcGIS Online is 15.325 credits.
Click Run.

Implement user authentication to access the spatial analysis service.
Define the parameters of the request.
Execute the operation. Note: This is a long transaction managed with a job request.
Handle the results.

APIs

ArcGIS API for Python

ArcGIS REST JS

Use dark colors for code blocks

Copy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

from arcgis import GIS
from arcgis.features.analysis import calculate_density

portal = GIS(username="<USERNAME>", password="<PASSWORD>")

parking_citations = "https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/Driveway_and_sidewalk_parking_violations/FeatureServer/0"

sf_boundary = "https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/SF_Boundary/FeatureServer/0"

results = calculate_density(
    input_layer=parking_citations,
    bounding_polygon_layer=sf_boundary,
    area_units="SquareMiles",
    classification_type="NaturalBreaks",
    num_classes=10,
    # output_name="<OUTPUT NAME>" if you want to save the results as a hosted feature layer.
)

result_features = results.query()

print(f"The density layer has {len(result_features.features)} new records")
map_widget = portal.map("San Francisco, CA")
map_widget.add_layer(result_features)
map_widget

Expand

Go to API reference

Service requests

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST arcgis.com/sharing/rest/portals/self HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
{
  "helperServices": {
    // Other parameters
    "analysis": {
      "url": "https://<YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer"
    },
    "geoenrichment": {
      "url": "https://geoenrich.arcgis.com/arcgis/rest/services/World/GeoenrichmentServer"
    }
  }
}

Request

HTTP

cURL

Use dark colors for code blocksCopy

POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/CalculateDensity/submitJob HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>
&inputLayer={"url":"https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/Driveway_and_sidewalk_parking_violations/FeatureServer/0","name":"Driveway_and_sidewalk_parking_violations"}
&areaUnits=SquareMiles
&boundingPolygonLayer={"url":"https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/SF_Boundary/FeatureServer/0","name":"SF_Boundary"}
&classificationType=NaturalBreaks
&numClasses=10
&outputName={"serviceProperties":{"name":"Density of parking tickets"}}
&context={}
&returnFeatureCollection=false

Response (JSON)

Use dark colors for code blocksCopy

{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobSubmitted",
  "results": {},
  "inputs": {},
  "messages": []
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/CalculateDensity/jobs/<JOB_ID> HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
// Executing job
{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobExecuting",
  "results": {},
  "inputs": {},
  "messages": []
}
// Job succeeded
{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobSucceeded",
  "results": {
      "resultLayer": {
          "paramUrl": "results/resultLayer"
      },
  },
  "inputs": {},
  "messages": []
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/CalculateDensity/jobs/<JOB_ID>/results/resultLayer HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&returnType=data
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
{
  "paramName": "resultLayer",
  "dataType": "GPString",
  "value": {
      "url": "<SERVICE_URL>",
      "itemId": "<ITEM_ID>"
  }
}

The analysis identified at least six areas where there is a higher density of parking violations within the limits of San Francisco (excluding Treasure Island and Yerba Buena Island).

Find hot spots

The Find Hot Spots (Gi* de Getis-Ord) analysis determines if there are any statistically significant clusters in the data. For a feature to be in a hot spot cluster, it must have a high value (in this instance a high number of parking violations) and be surrounded by other clusters with high values. For a feature to be in a cold spot cluster, the feature must have a low value and be surrounded by other clusters with low values.

To learn more, go to Find hot spots.

Steps to use the Map Viewer

Steps to use ArcGIS Python, REST JS, and REST APIs

In the Settings (light) toolbar, click Analysis > Tools > Find Hot Spots.
Set the following parameters:
- Input layer: Driveway and sidewalk parking violations.
- Variable type: Point counts.
- Aggregation shape type: Hexagon cells.
Set the Output name: Parking violation hot spots.
Click Estimate credits. The estimated cost for ArcGIS Location Platform is USD $1.53 and ArcGIS Online is 15.351 credits.
Click Run.
In the map, click on some hexagons to view the Number of Points and Statistical Significance in the popup.
Compare the results of the hot spot analysis to the Calculate density of incidents layer.

Implement user authentication to access the spatial analysis service.
Define the parameters of the request.
Execute the operation. Note: This is a long transaction managed with a job request.
Handle the results.

APIs

ArcGIS API for Python

ArcGIS REST JS

Expand

Use dark colors for code blocks

Copy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
from arcgis import GIS
from arcgis.features.analysis import find_hot_spots

portal = GIS(username="<USERNAME>", password="<PASSWORD>")

traffic_citations = "https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/Driveway_and_sidewalk_parking_violations/FeatureServer/0"
hexagon = "hexagon"

results = find_hot_spots(
    analysis_layer=traffic_citations,
    shape_type=hexagon,
    output_name="Parking citation hot spots",
)

result_features = results.layers[0].query()

print(f"The hot spot layer has {len(result_features.features)} new records")

map_widget = portal.map("San Francisco, CA")
map_widget.add_layer(results)
map_widget

Expand

Go to API reference

Service requests

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST arcgis.com/sharing/rest/portals/self HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
{
  "helperServices": {
    // Other parameters
    "analysis": {
      "url": "https://<YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer"
    },
    "geoenrichment": {
      "url": "https://geoenrich.arcgis.com/arcgis/rest/services/World/GeoenrichmentServer"
    }
  }
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/FindHotSpots/submitJob HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>
&analysisLayer={"url":"https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/Driveway_and_sidewalk_parking_violations/FeatureServer/0","name":"Driveway_and_sidewalk_parking_violations"}
&shapeType=hexagon
&returnProcessInfo=true
&outputName={"serviceProperties":{"name":"Parking incident hot spots"}}
&context={}

Response (JSON)

Use dark colors for code blocksCopy

{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobSubmitted",
  "results": {},
  "inputs": {},
  "messages": []
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/FindHotSpots/jobs/<JOB_ID> HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
// Executing job
{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobExecuting",
  "results": {},
  "inputs": {},
  "messages": []
}
// Job succeeded
{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobSucceeded",
  "results": {
      "hotSpotsResultLayer": {
          "paramUrl": "results/hotSpotsResultLayer"
      },
      "processInfo": {
          "paramUrl": "results/processInfo"
      }
  },
  "inputs": {},
  "messages": []
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/FindHotSpots/jobs/<JOB_ID>/results/hotSpotsResultLayer HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&returnType=data
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
{
  "paramName": "hotSpotsResultLayer",
  "dataType": "GPString",
  "value": {
      "url": "<SERVICE_URL>",
      "itemId": "<ITEM_ID>"
  }
}

In the results, the red hexagons represent clusters that are hot spots with a statistically high number of reported incidents and are surrounded by other clusters with a statistically high number of incidents. The blue hexagons represent clusters that are cold spots with a low number of incidents and are surrounded by other clusters with a low number of incidents.

Find outliers

The Find outliers (Anselin local Moran's I) analysis identifies both clusters and anomalous values (outliers). It determines whether the attribute value for each feature is significantly different, defined as the resultant z-score and p-value, from its neighbors. It will classify the feature as: High-High (a high value surrounded by other high values), High-Low (a high value surrounded by low values), Low-High (a low value surrounded by high values), and Low-Low (a low value surrounded by other low values). A feature is part of a cluster when it has a similar value with its neighbors. A feature is considered an outlier when it has dissimilar values from its neighbors.

To learn more, go to Find outliers.

Steps to use the Map Viewer

Steps to use ArcGIS Python, REST JS, and REST APIs

In the Settings (light) toolbar, click Analysis > Tools > Find Outliers.
Set the following parameters:
- Input layer: Driveway and sidewalk parking violations.
- Variable type: Point counts.
- Aggregation shape type: Hexagon cells
- Output name: Parking violation outliers.
Click Estimate credits. The estimated cost for ArcGIS Location Platform is USD $1.53 and ArcGIS Online is 15.351 credits.
Click Run.
In the map, click on some hexagons to view the Number of Points and Cluster/Outlier Type in the popup.
Compare the results of the analysis with the Calculate density of incidents and Parking violation hot spots layers.
In the Content (dark) toolbar, click Save and open > Save to save your web map.

Implement user authentication to access the spatial analysis service.
Define the parameters of the request.
Execute the operation. Note: This is a long transaction managed with a job request.
Handle the results.

APIs

ArcGIS API for Python

ArcGIS REST JS

Expand

Use dark colors for code blocks

Copy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
from arcgis import GIS
from arcgis.features.analysis import find_outliers

portal = GIS(username="<USERNAME>", password="<PASSWORD>")

traffic_citations = "https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/Driveway_and_sidewalk_parking_violations/FeatureServer/0"

results = find_outliers(
    analysis_layer=traffic_citations,
    shape_type="hexagon",
    output_name="Parking citation outliers"
)

result_features = results["outliers_result_layer"].layers[0].query()

print(
    f"The find outliers layer has {len(result_features.features)} new records"
)
map_widget = portal.map("San Francisco, CA")
map_widget.add_layer(results)
map_widget

Go to API reference

Service requests

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST arcgis.com/sharing/rest/portals/self HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
{
  "helperServices": {
    // Other parameters
    "analysis": {
      "url": "https://<YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer"
    },
    "geoenrichment": {
      "url": "https://geoenrich.arcgis.com/arcgis/rest/services/World/GeoenrichmentServer"
    }
  }
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/FindOutliers/submitJob HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>
&analysisLayer={"url":"https://services3.arcgis.com/GVgbJbqm8hXASVYi/arcgis/rest/services/Driveway_and_sidewalk_parking_violations/FeatureServer/0","name":"Driveway_and_sidewalk_parking_violations"}
&shapeType=hexagon
&returnProcessInfo=true
&outputName={"serviceProperties":{"name":"Parking incident outliers}}
&context={}

Response (JSON)

Use dark colors for code blocksCopy

{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobSubmitted",
  "results": {},
  "inputs": {},
  "messages": []
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/FindOutliers/jobs/<JOB_ID> HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
// Executing job
{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobExecuting",
  "results": {},
  "inputs": {},
  "messages": []
}
// Job succeeded
{
  "jobId": "<JOB_ID>",
  "jobStatus": "esriJobSucceeded",
  "results": {
      "outliersResultLayer": {
          "paramUrl": "results/outliersResultLayer"
      },
      "processInfo": {
          "paramUrl": "results/processInfo"
      }
  },
  "inputs": {},
  "messages": []
}

Request

HTTP

cURL

Use dark colors for code blocksCopy
POST <YOUR_ANALYSIS_SERVICE>/arcgis/rest/services/tasks/GPServer/FindOutliers/jobs/<JOB_ID>/results/outliersResultLayer HTTP/1.1
Content-Type: application/x-www-form-urlencoded

&f=json
&returnType=data
&token=<ACCESS_TOKEN>

Response (JSON)

Use dark colors for code blocksCopy
{
  "paramName": "outliersResultLayer",
  "dataType": "GPString",
  "value": {
      "url": "<SERVICE_URL>",
      "itemId": "<ITEM_ID>"
  }
}

The results show where there are anomalous values. Dark blue indicates significantly lower rates of parking violations as compared to its neighbors. Red indicates a higher rate of parking violations as compared to its neighbors.

What's next?

You have completed three different types of pattern analyses by calculating the density, finding hot spots, and finding the outliers of parking violations in San Francisco. Your map should look something like this.

Learn how to use additional tools, APIs, and location services in these tutorials:

Summarize data

Aggregate and summarize features using summarize analysis operations.

Map Viewer Python API REST JS REST API

Combine data

Overlay, join, and dissolve features using combine analysis operations.

Map Viewer Python API REST JS REST API

Find and extract data

Find data with attribute and spatial queries using find analysis operations.

Map Viewer Python API REST JS REST API