A feature service is a data service that stores and hosts spatial and nonspatial data online. In a feature service, spatial datasets are feature layers and non-spatial datasets are tables. You can query, edit and analyze data in a feature service. Feature service URLs are typically in the formats shown below.
URL for a feature service layer or table hosted in ArcGIS Online:
https
URL for a feature service layer or table hosted in ArcGIS Enterprise:
https
.
Refer to the feature service documentation for ArcGIS Online and ArcGIS Enterprise for more details.
Register a GIS
Authentication will be required when reading or saving a secured feature service hosted by ArcGIS Online or ArcGIS Enterprise.
GeoAnalytics Engine supports authentication by registering a GIS (i.e. a connection to ArcGIS Online or ArcGIS Enterprise)
with the register
function.
The default GIS in register
is ArcGIS Online. The following example shows registering ArcGIS Online as a
GIS with a username and password for a built-in account.
import geoanalytics
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")
To connect to ArcGIS Enterprise, specify the portal URL as shown in the following example for a built-in account.
import geoanalytics
geoanalytics.register_gis("myPortal", "https://example.com/portal", username="User", password="p@ssw0rd")
Each GIS name can only be registered once. To remove a GIS, use the unregister
function as shown in the
following example.
import geoanalytics
geoanalytics.unregister_gis("myGIS")
In addition to built-in accounts, the following authentication schemes are supported:
- Web-tier authentication secured with PKI
- User authentication with OAuth 2.0
Web-tier authentication secured with PKI
To connect to a PKI protected ArcGIS Enterprise, you can use a PKCS12 formatted certificate file (for example .pfx or .p12) and password when logging into your ArcGIS Enterprise using PKI-based client certificate authentication.
geoanalytics.register_gis("myPortal", "https://example.com/portal",
cert_file=r"\\path\to\mycert.pfx", cert_password="p@ssw0rd")
User authentication with OAuth 2.0
GeoAnalytics Engine supports OAuth 2.0 as an authentication method and acts as a serverless native application when using OAuth 2.0 authorization with ArcGIS Online or ArcGIS Enterprise. This mode of authorization requires a client id. If you don't have a client id, follow the steps below to obtain one by registering a new application with your GIS. Only one client id is required, so if your GIS has one already, you may use it instead of creating a new application.
- Log in to your ArcGIS Online or ArcGIS Enterprise organization.
- Go to the Content tab.
- Click New item, then select Application.
- On the New item dialog, select Other application and click Next.
- Type in a title for the application item. Optionally, specify the folder, tags, and summary.
- Click Save to add the item.
- On the item details page of this newly created application, browse to the Credentials section and find the Client ID. The string listed is the value that you will pass in as the client_id parameter when registering a GIS
You can register an OAuth 2.0 protected GIS with your client id and an authorization code. To generate an
authorization code, call register
with only the name
, url
, and client
parameters specified.
geoanalytics.register_gis("myPortal", "https://example.com/portal", client_id="drg5d5r575rfb")
IllegalArgumentException: 'authorization_code' is required when 'client_id' is provided. You can generate an
authorization code through the following URL: ...
Open the url provided in the exception in a browser (you may need to sign in) and copy the authorization code. You can now register your GIS as shown below.
geoanalytics.register_gis("myPortal", "https://example.com/portal",
client_id="drg5d5r575rfb", authorization_code="3464736")
Read from feature services
GeoAnalytics Engine supports loading data from feature services into a Spark DataFrame for use with any operations supported on a DataFrame in GeoAnalytics Engine or PySpark.
The following example shows the Python syntax for loading a feature service into a Spark DataFrame, where URL
is the
URL to a feature service layer or table. URL
is not required if specified using the DataFrameReader option listed in
the table below.
spark.read.format("feature-service").option().load(URL)
Below is a table of options that GeoAnalytics Engine supports when loading a feature service with Spark Data
.
DataFrameReader option | Example | Description |
---|---|---|
url | .option("url", "https | The URL of the feature service layer. |
gis | .option("gis", "my | Reference to a registered GIS. Required for secured feature services if token is not provided. |
token | .option("token", "my | ArcGIS Online or portal token. Required for secured feature services if gis is not provided. |
timeout | .option("timeout", 60) | Timeout in seconds for loading a feature service. |
connect | .option("connect | Timeout in seconds for connecting to the GIS. |
num | .option("num | The total number of retries that will be made after failing to read from a feature service layer. The default is 2. |
extent | .option("extent", "-90.0, 30.0, 90, 50") | Filters the feature service layer by the spatial extent specified using the format " . Values should be defined in the same units as the spatial reference of the feature layer. |
When loading a public feature service layer or table, you can pass the URL without options 'gis' or 'token':
spark.read.format("feature-service").load(URL)
When loading a secured feature service layer or table, a GIS connection or token will be required. For example:
import geoanalytics
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")
spark.read.format("feature-service").option("gis", "myGIS").load(URL)
For more examples on how to load a feature service layer, refer to the Read from feature services tutorial.
Write to feature services
GeoAnalytics Engine extends Spark and supports saving data in feature services. You can serve and share feature services over the internet. You can also access, visualize, or edit the published feature services in an ArcGIS Online or Enterprise portal.
The following script shows the Python syntax of saving feature services that GeoAnalytics Engine currently supports:
df.write.format("feature-service").option().save()
Below is a table of options and modes that GeoAnalytics Engine supports when saving a feature service with
Spark Data
. The default mode is to create a new feature service.
DataFrameReader option | Example | Description |
---|---|---|
append | .mode("append") | Append to existing layer in a feature service. |
overwrite | .mode("overwrite") | Overwrite existing layer in a feature service. |
gis | .option("gis", "my | Reference to a registered GIS. Required if token is not provided. |
token | .option("token", my | ArcGIS Online or portal token. Required if gis is not provided. |
portal | .option("portal | The Arcgis Online or Portal URL for saving new feature services. Required if token is provided. |
service | .option("service | The name of the feature service that will be created if writing to a new feature service. |
layer | .option("layer | The ID of the layer within a feature service. Required when using append or overwrite mode if layer is not provided. |
layer | .option("layer | The name of the layer within a feature service. Required when using append or overwrite mode if layer is not provided. |
service | .option("service | The URL of the feature service to append or overwrite to. Required when using append or overwrite mode. |
path | .option("path", "new | Alternative method for specifying the layer . |
truncate | .option("truncate", True) | Remove all records in the layer before appending. The default is False . |
max | .option("max | Maximum number of writers that will be used asynchronously when writing to a layer. The default is 12. A larger number can be specified if the data store you're writing to has the capacity to process more asynchronous requests. |
description | .option("description", "my gis data") | Item description for a new feature service. |
tag | .option("tag", "flood") | Item tags for a new feature service. |
snippet | .option("snippet", "flood") | Item snippet for a new feature service. |
datasource | .option("datasource | Optional when writing to ArcGIS Enterprise. Choose from spatiotemporal or relational for the data store type. The default is relational . |
max | .option("max | Specify the maximum allowed length for a string field. |
rename | .option("rename | Rename the field by replacing the space in the field name with an underscore. The default is False . |
upsert | .option("upsert | The layer field to be used when matching features with upsert. Any field that has a unique index can be used. When writing to a new layer and specifying the upsert , a unique index will be created on the specified field. |
When saving a feature service, either a registered GIS or a token is required. Below is an example of registering a GIS and saving a feature service.
import geoanalytics
geoanalytics.register_gis("myGIS", username="User", password="p@ssw0rd")
data = [
('{"x": -0.126,"y": 51.504,"m": 187.73}', 43.26),
('{"x": -0.126,"y": 51.504}', 97.22)
]
df = spark.createDataFrame(data, ["esri_json", "value"]) \
.withColumn("point", ST.point_from_esri_json("esri_json", 4326))
df.write.format("feature-service") \
.option("gis", "myGIS") \
.option("serviceName", "esri_json_output") \
.save()
For more examples on how to save feature services, refer to the Write to feature services tutorial.
Usage notes
- A layer or table ID must be included in the feature service URL when loading.
- Secured feature services can be read by either providing a registered GIS or a token.
- When loading a feature service layer with geometries, the geometry field will be included in the result DataFrame automatically as as a point, multipoint, line, or polygon column.
- A registered GIS or a token and Portal URL are required when saving a feature service.
- When saving a Spark DataFrame with a geometry column, a feature layer will be created. For a Spark DataFrame without a geometry column, a table will be created in the new feature service. A spatial reference is required to be set on the geometry column when saving to a feature layer.
- The generic
geometry
type is not supported when saving a feature service. The geometry type should be one of the following types:point
,multipoint
,line
, orpolygon
. - Any 64-bit signed integer columns (
Long
in Spark) will be cast toType Double
and written to feature service asType esri
.Field Type Double - Beginning with version 1.2.0,
esri
fields can be read from feature services asField Type Big Integer Long
columns. ReadingType esri
fields is not supported in previous versions.Field Type Big Integer - The option
layer
can only be used to overwrite or append to an existing layer. Use theId layer
option when creating a new layer.Name - The option
service
should only be used when writing to a new feature service, and the service name needs to be unique within your ArcGIS Enterprise or ArcGIS Online organization. To overwrite or append to an existing layer, useName service
.Url - When providing
service
,Url layer
, orId layer
must also be specified.Name - When saving a feature service, the
has
andZ has
properties will always be set toM false
.