Summarize within

Summarize Within calculates statistics in areas where geometries from the input DataFrame are within or overlap specified boundaries. The boundaries can be a polygon geometry column or you can summarize within hexagonal or square bins.

Summarize Within workflow

Examples of summarizing points within polygons (first row), linestrings within polygons (second row), and polygons within polygons (third row).

Usage notes

  • Summarize Within takes a DataFrame and a boundary, and stacks them on top of each other. After stacking, you can look down through the stack and count the number of summarized input records that fall within the input boundaries. You can also calculate statistics about the attributes of the input summary records, such as minimum, maximum, sum, mean, weighted mean, and so on.

  • There are two ways to specify the boundaries:

    • Use a polygon DataFrame by specifying setSummaryPolygons().
    • Use a square or hexagonal bin of a specified size that is generated when the analysis is run by specifying setSummaryBins().
  • The bin size specifies how large the bins are. If you are summarizing into hexagons, the size is the height of each hexagon, and the radius of the resulting hexagon will be the height divided by the square root of three. If you are aggregating into squares, the bin size is the height of the square, which is equal to the width.

    ap measure
  • Use Summarize Within to calculate standard statistics as well as geographically weighted statistics. Standard statistics summarize the statistical values without weighting. Weighted statistics calculate values using the geographically weighted attributes of lines within a polygon, or the attributes of polygons within a polygon. Weighted statistics do not apply to points within polygons.

  • Standard statistics and geographically weighted statistics can be calculated for attributes that represent either counts or rates. These are defined as follows:

    • Counts—Attributes that represent a sum or quantity of an entity at a point location, along a line, or within a polygon. Examples of count-type attributes include the population of a country, the number of taxi pickups in a census block, and the number of dams along a river. For line and polygon features, counts are proportioned before calculating standard or weighted statistics.
    • Rates—Attributes that represent a ratio or index at a point location, along a line, or within a polygon. Examples of rate-type attributes include the population density of a country, the speed limit of a road, or the walkability score of a neighborhood. Rates are never proportioned.

    For count-type attributes, values are proportioned according to the amount of the line within a polygon or the amount of the polygon within another polygon prior to calculating statistics. Statistics are calculated the same way for count-type and rate-type attributes when the summary features are points.

  • You can calculate the lengths and areas of the summarized geometries within each polygon using the options in the table below. Options are based on the geometry of the summarized DataFrame.

Input geometryDescriptionOption
PointsThe count of summarized points within each boundary.None
LinestringsThe length of summarized linestrings within or intersecting each boundary.
  • Miles
  • Yards
  • Feet
  • Kilometers
  • Meters
PolygonsThe area of summarized polygons within or intersecting each boundary.
  • Square Miles
  • Square Yards
  • Square Feet
  • Square Kilometers
  • Square Meters
  • Hectares
  • Acres
  • For standard statistics, there are ten options: count, sum, mean, minimum, maximum, range, standard deviation, variance, first, and last. Count and sum will not be calculated for rate-type attributes. There are four options for string statistics: count, any, first, and last.

  • For weighted statistics, there are three options: mean, standard deviation, and variance. Weighted statistics are not calculated for string data.

  • To calculate first or last, time needs to be enabled on the input DataFrame.

    Learn more about enabling time on a DataFrame.

  • Analysis with binning requires that your input DataFrame's geometry has a projected coordinate system. If your data is not in a projected coordinate system, the tool will transform your input to a World Cylindrical Equal Area (SRID: 54034) projection. You can transform your data to a projected coordinate system by using ST_Transform.

    Learn more about coordinate systems and transformations

  • Optionally, specify a field name using setGroupBy() so statistics are calculated separately for each unique field value. When a group by field value is specified, a summary table listing each record and statistic is also created.

  • The options include_minor_major_fields and include_group_percentages are part of the group by option (setGroupBy()). The minority and majority will be the least and most dominant value from the group field, respectively, where dominance is determined using the count of points, total length, or total area of each value.

  • When the specified value for include_minor_major_fields is True, two fields will be added to the result DataFrame. The fields will list the values from the group field that are the minority and majority for each result.

  • The include_group_percentages option can only be used when you specify a value of True for include_minor_major_fields. When the value specified for include_group_percentages is True, two fields will be added to the result DataFrame listing the percentage of the count of points, total length, or total area that belong to the minority and majority values for each input record. A percentage field will also be added to the result table listing the percentage of the count of points, total length, or total area that belong to all values from the group by field for each input record.

  • The output DataFrame always contains polygons. Only polygons that intersect the summarized geometries will be returned. Other polygons will be completely removed from the result.

    ap point in poly

Limitations

  • You can only calculate statistics on the records that intersect your boundary.

Results

The following fields are included in the output polygon DataFrame:

FieldDescription
bin_geometryThe result bin geometries.
countThe count of summarized records that intersect each boundary.
sum_length_<linearunit>, or sum_area_<areaunit>The total length of linestrings within or intersecting the boundary or total area of summarized polygons within or intersecting a polygon. These values are returned when you specify a value of True for includeShapeSummary() and are returned in the specified unit.
<statistic>_<fieldname>Specified standard statistics will each create a field named in the following format: <statistic>_<fieldname>. For example, the maximum and standard deviation of the field id is MAX_id and SD_id.
p<statistic>_<fieldname>Specified weighted statistics will each create a field named in the following format: p<statistic>_<fieldname>. For example, the mean and standard deviation of the field pop is pMEAN_pop and pSD_pop.
minority_<fieldname>This value is returned when you create a group-by table and specify minority and majority calculations. This represents the values for the specified field that is the minority in each polygon. For example, there are five points within a polygon with a field called color and values of red, blue, blue, green, green. If you create a group by the color field, the value for the minority_color field is red.
majority_<fieldname>This value is returned when you create a group-by table and specify minority and majority calculations. This represents the values for the specified field that is the majority in each polygon. For example, there are five points within a polygon with a field called color and values of red, blue, blue, green, green. If you create a group by the color field, the value for the majority_color field is blue;green.
minority_<fieldname>_percent This value is returned when you create a group-by table and specify percent shapes. This represents the percentages of the count for the specified field that is the minority in each polygon. For example, there are five points within a polygon with a field called color and values of red, blue, blue, green, green. If you create a group by the color field, the value for the minority_color_percent field is 20 (calculated as 1/5).
majority_<fieldname>_percentThis value is returned when you create a group-by table and specify percent shapes. This represents the percentages of the count for the specified field that is the majority in each polygon. For example, there are five points within a polygon with a field called color and values of red, blue, blue, green, green. If you create a group by the color field, the value for the majority_color_percent field is 40 (calculated as 2/5).
join_id This value is returned when you create a group-by table. This is an ID to link records to the group-by table. Every join_id field corresponds to one or more records in the group-by table.

The following fields are included in the output group-by DataFrame:

FieldDescription
join_idThis is an ID to link records to the polygon DataFrame. Each polygon will have one or more records with the same ID that represent all of the group-by values. For example, there are five points within a polygon with a field called color and values of red, blue, blue, green, green. The group-by table will have three records representing that polygon (same join ID), one for each of the colors red, blue, and green.
countThe count of the specified group within the joined polygon. For example, red is 1 for the selected polygon.
<statistic>_<fieldname>Any specified statistic calculated for each group.
percentcountThe percentage each group contributes to the total count in the polygon. Using the above example, red contributes 1/5 = 20, blue contributes 2/5 = 40, and green contributes 2/5 = 20.

Performance notes

Improve the performance of Summarize Within by doing one or more of the following:

  • Only analyze the records in your area of interest. You can pick the records of interest by using one of the following SQL functions:

    • ST_Intersection—Clip to an area of interest represented by a polygon. This will modify your input records.
    • ST_BboxIntersects—Select records that intersect an envelope.
    • ST_EnvIntersects—Select records having an evelope that intersects the envelope of another geometry.
    • ST_Intersects—Select records that intersect another dataset or area of intersect represented by a polygon.
  • If you are using bins, larger bins will perform better than smaller bins. If you are unsure which size to use, start with a larger bin to prototype.

Similar capabilities

Syntax

For more details, go to the GeoAnalytics Engine API reference for summarize within.

SetterDescriptionRequired
addRateField(rate_field)Adds a field in the input DataFrame to a list of fields that represent rates, indices, or ratios. Examples of rate-type attributes include the population density of a country, the speed limit of a road, or the walkability score of a neighborhood. Rates are never proportioned. By default all fields are assumed to represent counts or amounts and will be proportioned.No
addStandardSummaryField(summary_field, statistic, alias=None)Adds a standard summary statistic of a field in the input DataFrame to the result DataFrame. Statistics for numeric fields include Count, Sum, Mean, Max, Min, Range, Stddev, Var, First, Last, or Any. Count and sum will not be calculated for rate-type fields. There are four options for string statistics: Count, Any, First, and Last.No
addWeightedSummaryField(summary_field, statistic, alias=None)Adds a weighted summary statistic of a field in the input DataFrame to the result DataFrame. Statistics include Mean, Stddev, and Var. Weighted statistics are not calculated for string fields.No
includeShapeSummary(include=True, units=None)Sets to the tool to calculate statistics based on the geometry type of the primary geometry column in the input DataFrame, such as the length of lines or areas of polygons within each summary polygon.No
run(dataframe)Runs the Summarize Within tool using the provided DataFrame.Yes
setGroupBy(group_by_field, include_minor_major_fields=True, include_group_percentages=True)Sets a field from the input DataFrame that will be used to calculate statistics for each unique value. When setGroupBy() is called, the tool will return a DataFrame containing the grouped statistics in addition to a DataFrame containing the summaries.No
setSummaryBins(bin_size, bin_size_unit, bin_type='square')Sets the size and shape of bins that the input DataFrame will be summarized into.One of setSummaryBins() or setSummaryPolygons() is required.
setSummaryPolygons(summary_polygons)Sets the DataFrame containing a column of polygons that the input DataFrame will be summarized into.One of setSummaryBins() or setSummaryPolygons() is required.

Examples

Run Summarize Within

Python
Use dark colors for code blocksCopy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# Log in
import geoanalytics
geoanalytics.auth(username="myusername", password="mypassword")

# Imports
from geoanalytics.tools import SummarizeWithin
from geoanalytics.tools import ReconstructTracks
from geoanalytics.sql import functions as ST

# Path to the hurricane tracks dataset
hurricanes_data_path = r"https://services2.arcgis.com/FiaPA4ga0iQKduv3/arcgis/rest/" \
            "services/IBTrACS_ALL_list_v04r00_points_1/FeatureServer/0"

# Create a hurricanes tracks DataFrame and filter to a smaller extennt of area
hurricanes_df = spark.read.format("feature-service").load(hurricanes_data_path) \
    .withColumn("bbox_intersects", ST.bbox_intersects("shape",-10512137.72, -9527997.38,3278846.39,4303954.46)) \
    .where("bbox_intersects == 'true'") \
    .where("BASIN == 'NA'")

# Use Reconstruct Tracks to create hurricane paths
rt_result = ReconstructTracks() \
            .setTrackFields("NAME") \
            .setDistanceMethod(distance_method="Planar") \
            .run(dataframe=hurricanes_df)

# Use Summarize Within to summarize hurricane tracks into bins to
#     visualize a track heat map
result = SummarizeWithin() \
           .setSummaryBins(bin_size=200, bin_size_unit="Kilometers", bin_type='hexagon') \
           .includeShapeSummary(include=True, units="Kilometers") \
           .run(dataframe=rt_result)

Plot results

Python
Use dark colors for code blocksCopy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# Plot the summarized result with shorelines (continent outlines) near Florida
continents_path = "https://services.arcgis.com/P3ePLMYs2RVChkJx/ArcGIS/rest/" \
    "services/World_Continents/FeatureServer/0"
shoreline_df = spark.read.format("feature-service").load(continents_path)

result_plot = result.output.st.plot(cmap_values="COUNT",
                                    cmap="coolwarm",
                                    figsize=(14,8),
                                    legend=True,
                                    basemap="light")

shoreline_plot = shoreline_df.st.plot(edgecolors="black",
                                    alpha=0.3,
                                    figsize=(14,8),ax=result_plot)


result_plot.set_title("Hurricane track heat map near Florida and the Gulf of Mexico")
result_plot.set_xlabel("X (Meters)")
result_plot.set_ylabel("Y (Meters)");
result_plot.set_xlim(left=-10512137, right=-8027997)
result_plot.set_ylim(bottom=2688846, top=4303954)
Plotting example for a Summarize Within result. Hurricane tracks summarized into hexagon bins is shown.

Version table

ReleaseNotes

1.0.0

Tool introduced

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.