Introduction to the Spatially Enabled DataFrame

The Spatially Enabled DataFrame (SEDF) creates a simple, intutive object that can easily manipulate geometric and attribute data.

New at version 1.5, the Spatially Enabled DataFrame is an evolution of the SpatialDataFrame object that you may be familiar with. While the SDF object is still avialable for use, the team has stopped active development of it and is promoting the use of this new Spatially Enabled DataFrame pattern. The SEDF provides you better memory management, ability to handle larger datasets and is the pattern that Pandas advocates as the path forward.

The Spatially Enabled DataFrame inserts a custom namespace called spatial into the popular Pandas DataFrame structure to give it spatial abilities. This allows you to use intutive, pandorable operations on both the attribute and spatial columns. Thus, the SEDF is based on data structures inherently suited to data analysis, with natural operations for the filtering and inspecting of subsets of values which are fundamental to statistical and geographic manipulations.

The dataframe reads from many sources, including shapefiles, Pandas DataFrames, feature classes, GeoJSON, and Feature Layers.

This document outlines some fundamentals of using the Spatially Enabled DataFrame object for working with GIS data.

import pandas as pd
from arcgis.features import GeoAccessor, GeoSeriesAccessor

Accessing GIS data

GIS users need to work with both published layers on remote servers (web layers) and local data, but the ability to manipulate these datasets without permanently copying the data is lacking. The Spatial Enabled DataFrame solves this problem because it is an in-memory object that can read, write and manipulate geospatial data.

The SEDF integrates with Esri's ArcPy site-package as well as the open source pyshp, shapely and fiona packages. This means the ArcGIS API for Python SEDF can use either of these geometry engines to provide you options for easily working with geospatial data regardless of your platform. The SEDF transforms data into the formats you desire so you can use Python functionality to analyze and visualize geographic information.

Data can be read and scripted to automate workflows and just as easily visualized on maps in Jupyter notebooks. The SEDF can export data as feature classes or publish them directly to servers for sharing according to your needs. Let's explore some of the different options available with the versatile Spatial Enabled DataFrame namespaces:

Reading Web Layers

Feature layers hosted on ArcGIS Online or ArcGIS Enterprise can be easily read into a Spatially Enabled DataFrame using the from_layer method. Once you read it into a SEDF object, you can create reports, manipulate the data, or convert it to a form that is comfortable and makes sense for its intended purpose.

Example: Retrieving an ArcGIS Online item and using the layers property to inspect the first 5 records of the layer

from arcgis import GIS
gis = GIS()
item = gis.content.get("85d0ca4ea1ca4b9abf0c51b9bd34de2e")
flayer = item.layers[0]

# create a Spatially Enabled DataFrame object
sdf = pd.DataFrame.spatial.from_layer(flayer)
sdf.head()

When you inspect the type of the object, you get back a standard pandas DataFrame object. However, this object now has an additional SHAPE column that allows you to perform geometric operations. In other words, this DataFrame is now geo-aware.

type(sdf)

pandas.core.frame.DataFrame

Further, the DataFrame has a new spatial property that provides a list of geoprocessing operations that can be performed on the object. The rest of the guides in this section go into details of how to use these functionalities. So, sit tight.

Reading Feature Layer Data

As seen above, the SEDF can consume a Feature Layer served from either ArcGIS Online or ArcGIS Enterprise orgs. Let's take a step-by-step approach to break down the notebook cell above and then extract a subset of records from the feature layer.

Example: Examining Feature Layer content

Use the from_layer method on the SEDF to instantiate a data frame from an item's layer and inspect the first 5 records.

# Retrieve an item from ArcGIS Online from a known ID value
known_item = gis.content.get("85d0ca4ea1ca4b9abf0c51b9bd34de2e")
known_item

USA Major Cities
This layer presents the locations of cities within the United States with populations of approximately 10,000 or greater, all state capitals, and the national capital.Feature Layer Collection by esri_dm
Last Modified: August 22, 2019
0 comments, 1,839,428 views

# Obtain the first feature layer from the item
fl = known_item.layers[0]

# Use the `from_layer` static method in the 'spatial' namespace on the Pandas' DataFrame
sdf = pd.DataFrame.spatial.from_layer(fl)

# Return the first 5 records. 
sdf.head()

NOTE: See Pandas DataFrame head() method documentation for details.

You can also use sql queries to return a subset of records by leveraging the ArcGIS API for Python's Feature Layer object itself. When you run a query() on a FeatureLayer, you get back a FeatureSet object. Calling the sdf property of the FeatureSet returns a Spatially Enabled DataFrame object. We then use the data frame's head() method to return the first 5 records and a subset of columns from the DataFrame:

Example: Feature Layer Query Results to a Spatially Enabled DataFrame

We'll use the AGE_45_54 column to query the data frame and return a new DataFrame with a subset of records. We can use the built-in zip() function to print the data frame attribute field names, and then use data frame syntax to view specific attribute fields in the output:

# Filter feature layer records with a sql query. 
# See https://developers.arcgis.com/rest/services-reference/query-feature-service-layer-.htm

df = fl.query(where="AGE_45_54 < 1500").sdf

for a,b,c,d in zip(df.columns[::4], df.columns[1::4],df.columns[2::4], df.columns[3::4]):
    print("{:<30}{:<30}{:<30}{:<}".format(a,b,c,d))

AGE_10_14                     AGE_15_19                     AGE_20_24                     AGE_25_34
AGE_35_44                     AGE_45_54                     AGE_55_64                     AGE_5_9
AGE_65_74                     AGE_75_84                     AGE_85_UP                     AGE_UNDER5
AMERI_ES                      ASIAN                         AVE_FAM_SZ                    AVE_HH_SZ
BLACK                         CAPITAL                       CLASS                         FAMILIES
FEMALES                       FHH_CHILD                     FID                           HAWN_PI
HISPANIC                      HOUSEHOLDS                    HSEHLD_1_F                    HSEHLD_1_M
HSE_UNITS                     MALES                         MARHH_CHD                     MARHH_NO_C
MED_AGE                       MED_AGE_F                     MED_AGE_M                     MHH_CHILD
MULT_RACE                     NAME                          OBJECTID                      OTHER
OWNER_OCC                     PLACEFIPS                     POP2010                       POPULATION
POP_CLASS                     RENTER_OCC                    SHAPE                         ST

# Return a subset of columns on just the first 5 records
df[['NAME', 'AGE_45_54', 'POP2010']].head()

Accessing local GIS data

The SEDF can also access local geospatial data. Depending upon what Python modules you have installed, you'll have access to a wide range of functionality:

• If the ArcPy module is installed, meaning you have installed ArcGIS Pro and have installed the ArcGIS API for Python in that same environment, the DataFrame then has methods to read a subset of the ArcGIS Desktop supported geographic formats, most notably:
• feature classes
• shapefiles,
• ArcGIS Server Web Services and ArcGIS Online Hosted Feature Layers
• OGC Services
• If the ArcPy module is not installed, the SEDF from_featureclass method only supports consuming an Esri shapefile

  Please note that you must install the pyshp package to read shapefiles in environments that don't have access to ArcPy.

Example: Reading a Shapefile

You must authenticate to ArcGIS Online or ArcGIS Enterprise to use the from_featureclass() method to read a shapefile with a Python interpreter that does not have access to ArcPy.

g2 = GIS("https://www.arcgis.com", "username", "password")

g2 = GIS("https://pythonapi.playground.esri.com/portal", "arcgis_python", "amazing_arcgis_123")

sdf = pd.DataFrame.spatial.from_featureclass("path\to\your\data\census_example\cities.shp")
sdf.tail()

Example: Reading a Featureclass from FileGDB

You must have fiona installed if you use the from_featureclass() method to read a feature class from FileGDB with a Python interpreter that does not have access to ArcPy.

sdf = pd.DataFrame.spatial.from_featureclass("path\to\your\data\census_example\census.gdb\cities")
sdf.head()

Saving Spatially Enabled DataFrames

The SEDF can export data to various data formats for use in other applications.

Export Options

• Feature Layers
• Feature Collections
• Feature Set
• GeoJSON
• Feature Class
• Pickle
• HDF

Export to Feature Class

The SEDF allows for the export of whole datasets or partial datasets.

Example: Export a whole dataset to a shapefile:

sdf.spatial.to_featureclass(location=r"c:\output_examples\census.shp")

'c:\\output_examples\\census.shp'

The ArcGIS API for Python installs on all macOS and Linux machines, as well as those Windows machines not using Python interpreters that have access to ArcPy will only be able to write out to shapefile format with the to_featureclass method. Writing to file geodatabases requires the ArcPy site-package.

Example: Export dataset with a subset of columns and top 5 records to a shapefile:

for a,b,c,d in zip(sdf.columns[::4], sdf.columns[1::4], sdf.columns[2::4], sdf.columns[3::4]):
    print("{:<30}{:<30}{:<30}{:<}".format(a,b,c,d))

PLACENS                       GEOID                         NAMELSAD                      CLASSFP
FUNCSTAT                      ALAND                         AWATER                        INTPTLAT

columns = ['NAME', 'ST', 'CAPITAL', 'STFIPS', 'POP2000', 'POP2007', 'SHAPE']
sdf[columns].head().spatial.to_featureclass(location=r"/path/to/your/data/directory/sdf_head_output.shp")

'/path/to/your/data/directory/sdf_head_output.shp'

Example: Export dataset to a featureclass in FileGDB:

sdf.spatial.to_featureclass(location=r"c:\output_examples\census.gdb\cities");

Publish as a Feature Layer

The SEDF allows for the publishing of datasets as feature layers.

Example: Publishing as a feature layer:

lyr = sdf.spatial.to_featurelayer('census_cities', folder='census')
lyr

census_cities
Feature Layer Collection by api_data_owner
Last Modified: September 11, 2019
0 comments, 0 views