Cloning Content

Introduction

One request the Python API team has heard repeatedly from Web GIS administrators: What's the best way to move my content from a development Enterprise or organization to staging to production?

This Guide provides one possible roadmap for how to get content from one Web GIS to another - a template for demonstrating basic software concepts when transferring content so the workflow can be modified for other types of content and tailored for an administrator's particular need.

The clone_items() function on the ContentManger aims to create an exact duplicate of one item with all its uses and functionality from an original organizational deployment, whether ArcGIS Online, Enteprise, or Kubernetes to a second organizational deployment. The relationship between any supported, related, or resource items to the items being cloned must be considered when cloning an item.

When talking about cloning items, let's establish some basic terminology. Cloning can refer to the transfer of items between any organizational deployment, from either ArcGIS Enterprise, ArcGIS Online, or ArcGIS Enterprise for Kubernetes to any other deployment type. Rather than tangling ourselves up when trying to distinguish between these portal types while writing about them, we'll refer to organization where the content originates as the source regardless of deployment type. We'll consider the organization to which we'll clone content as the target.

This document illustrates the workflow of gathering source items using a source administrator account, and cloning them into a target administrator account. More specifically, this guide walks through an approach of cloning hosted feature layers and web maps. Web Maps can consume these layer items and their source services as operational layer building blocks. For information on each of these item types, please click the hyperlinks for each type.

This is an administrator workflow. Using administrator accounts guarantees all necessary privileges within the source to access the item and any dependencies and within the target to create the services and items. The target administrator subsequently can create user accounts and reassign item ownership and/or group membership according to need. This might also be the simplest workflow when the portal has users managed by an external identity store.

Supported Items

As originally written, Python API developers designed the clone_items() function for transferring the following item types:

• Hosted Web Applications built with Web AppBuilder or shared using Configurable App Templates
• Web Maps
• Hosted Feature Layers
• Hosted Feature Layer Views
• Feature Collections
• Survey123 Forms
• Workforce Projects
• StoryMaps
  • See Cloning Complex Items
• Operation Views
• Dashboards
  • See Cloning Complex Apps
• QuickCapture Projects
• ArcGIS Notebooks
• Simple Types
  • Those items with a download option (see Data files for items that may be in a Web GIS and available for download), including zipped file geodatabases and shapefiles, code samples, zip files, and packages amongst others.

clone_items() clones the dependencies for the more complex items listed above. For example, cloning an existing web application, clones the web map and all hosted feature layers referenced in the map.

clone_items() will not clone map services and image services. Since these services can be published to servers other than the hosted server in a configuration, it's impossible for the function to determine where to publish them in the target. As a result, these items will copy over, but will continue to point back to the original source URL.

Let's work through examples of cloning individual items and inspect the results.

First, let's import the necessary libraries and connect to our source and target GIS:

Import libraries

from pathlib import Path
import sys
import pandas as pd

from arcgis.gis import GIS, Item
from arcgis.env import active_gis
from arcgis.features import FeatureLayerCollection
from arcgis.mapping import WebMap

Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.

Connect to source and target portals

Let's start from a discovery position as an administrator. We'll simulate an administrator who's been testing and developing using an ArcGIS Online deployment as a development environment and wants to clone items into an on-premise ArcGIS Enterprise for further development.

source = GIS(profile="your_online_admin_profile")
print(source)

GIS @ https://geosaurus.maps.arcgis.com version:2023.3

target = GIS(profile="your_data_owner_profile")
print(target)

GIS @ https://example.url.com/portal version:10.3

Get started with cloning

Let's start with an immediate demonstration of what clone_items() can do. We're logged in as an administrator and we'll get() a specific hosted feature layer item owned by one of the users in the source. We'll then clone it into the target while utilizing the owner parameter to specify a particular User in the target to own the cloned content. We can quickly examine the resulting url of the cloned feature layer to confirm the new item.

hosted_flyr = source.content.get("3213ff60f81c46a0a970ec31dde368ac")
hosted_flyr

Arkansas Hospitals
This data provides the Hospital information for the State of Arkansas to track disease related data in a mapping environment, and to do so in a way that electronic map information can be transmitted quickly and efficiently to those in the public health network. The database provides location information for use in local and regional cartographic and spatial analysis applications. It is the intention of the Arkansas State Land Information Board to facilitate dissemination of public data.Feature Layer Collection by api_data_owner
Last Modified: January 25, 2024
0 comments, 105 views

hosted_flyr.url

'https://services7.arcgis.com/JEwYeAy2cc8qOe3o/arcgis/rest/services/Arkansas_Hospitals/FeatureServer'

cloned_flyr = target.content.clone_items(items=[hosted_flyr],
                                         owner="api_data_owner",
                                         folder="cloning_guide")

cloned_flyr[0]

Arkansas Hospitals
This data provides the Hospital information for the State of Arkansas to track disease related data in a mapping environment, and to do so in a way that electronic map information can be transmitted quickly and efficiently to those in the public health network. The database provides location information for use in local and regional cartographic and spatial analysis applications. It is the intention of the Arkansas State Land Information Board to facilitate dissemination of public data.Feature Layer Collection by api_data_owner
Last Modified: February 20, 2024
0 comments, 1 views

cloned_flyr[0].url

'https://pythonapi.playground.esri.com/server/rest/services/Hosted/Arkansas_Hospitals/FeatureServer'

We can see that the clone_items() function returns a list containing the cloned items. Indexing the list we can see the operation created a new hosted feature layer in the target organization owned by the api_data_owner user we entered as the owner argument.

Cloning multiple items simultaneously

We demonstrated above passing one item in the items parameter list. Now let's see how clone_items() will clone every item in the list. First let's search() the source function for items owned by a particular user and clone the list items into our target all at the same time.

tester_content = source.content.search(f"tags:disaster_testing AND owner:api_data_owner")
tester_content

[<Item title:"Earthquake Damage Map" type:Web Map owner:api_data_owner>,
 <Item title:"earthquakes_2" type:CSV owner:api_data_owner>,
 <Item title:"earthquakes_sample_data" type:CSV owner:api_data_owner>,
 <Item title:"earthquakes_2" type:Feature Layer Collection owner:api_data_owner>]

cloned_items = target.content.clone_items(items=tester_content, 
                                          folder=output_folder.name)

cloned_items

[<Item title:"earthquakes_2" type:Feature Layer Collection owner:api_data_owner>,
 <Item title:"earthquakes_2" type:CSV owner:api_data_owner>,
 <Item title:"earthquakes_sample_data" type:CSV owner:api_data_owner>,
 <Item title:"Earthquake Damage Map" type:Web Map owner:api_data_owner>]

We can see that the clone_items() function cloned all the items in the list we passed to the items argument. We can also see that by not passing in an owner argument, the items are owned by the logged in user that ran clone_items().

target.users.me.username

'api_data_owner'

There's a quick demonstration of how the clone_items() function replicates items from a source to a target. Now let's move onto iterating through a list of hosted feature layers and working with some additonal parameters to clone information products that consume them.

The cloning process

A very important parameter in the clone_items() function impacts its output:

• search_existing_items - The possible values are True or False.

The default value is True. Let's describe what happens when the value is set to True. When any source item is cloned into the target, the clone_items() function assigns the cloned item all the essential item type typeKeywords (See hyperlink for typeKeywordsautomatically assigned to items) plus an additional typeKeyword of source-<source_item_id_value>. For example, if clone_items() clones a feature layer item from source item item id d879c7d972b1d989b97d037c7a7737d6, the resulting feature layer item in the target will have a typeKeyword of source-d879c7d972b1d989b97d037c7a7737d6 in addition to all essential typeKeywords. Before any actual clone, clone_items() searches the target for any item with a typeKeyword matching that pattern and if it finds one uses the existing target item rather than cloning the item again.

If the argument is set to False, the specified item(s) and its dependent items will be cloned into the target no matter whether they already exist.

Web Maps

We can create a list of the web map items using the advanced_search() method. First we'll get a list of all items, and then convert it to a Pandas DataFrame to filter out for our web maps:

# Get all items owned by the user
owner_items = source.content.advanced_search(query=f"owner:{source.users.me.username}", 
                                             max_items=-1)["results"]

# Convert list to a Pandas DataFrame
owner_items_df = pd.DataFrame(owner_items)

Use Pandas groupby() to create individual group objects of each item type owned by the user. Then use the get_group() method to return all the Web Maps.

wm_item_df = owner_items_df.groupby("type").get_group("Web Map")
wm_item_df

The resulting DataFrame's index uses the position value of each web map in the original dataframe for the row index, so we'll reset it to a zero-based index:

wm_item_df.index

Index([1081, 1092, 1250, 1251, 1255, 1363, 1364, 1365, 1366, 1367, 1368, 1401,
       1402, 1414, 1428, 1454],
      dtype='int64')

web_maps = wm_item_df.reset_index(drop=True)

web_maps.index

RangeIndex(start=0, stop=16, step=1)

Let's retrieve a web map using a tag search so we can clone it into our target:

def check_wm(tag_list):
    return "power_plants" in tag_list

power_plant_df = web_maps[web_maps.tags.apply(check_wm)]

power_plant_df.index

Index([3], dtype='int64')

power_plant_df.loc[3].id

'6b1f7ea22e2543aa9849e34097d05570'

power_plant_wm_item = source.content.get(power_plant_df.loc[3].id)
power_plant_wm_item

US Power Plant Data
Web Map of US Power Plant Data.Web Map by ArcGISPyAPIBot
Last Modified: February 20, 2024
0 comments, 4 views

Let's use the WebMap object to print information about each of the layers in the Web Map. We'll look at the layer name and the url, specifically paying attention to the beginning of the path componenet of the url, which contains the organization id of the host of the layer.

wm_obj = WebMap(power_plant_wm_item)

for wm_lyr in wm_obj.definition["operationalLayers"]:
    print(f"{wm_lyr['title']}\n{' '*2}{wm_lyr['url']}")
    print(f"{' ' * 2}Host Organization id: {wm_lyr['url'].split('/')[3]}")
    print("\n")

US Power Plants
  https://services7.arcgis.com/JEwYeAy2cc8qOe3o/arcgis/rest/services/US_Power_Plants/FeatureServer/0
  Host Organization id: JEwYeAy2cc8qOe3o

Printing out our source organization id, we can see that the layer in the web map is hosted in the source.

print(f"{source.properties.id}")

JEwYeAy2cc8qOe3o

We can use the search_existing_items parameter to handle how we want the items consumed in our web map to behave upon cloning. Given what we've learned about this parameter, we can rely on it to detect whether any of the item ids consumed in our web maps match any of the typeKeywords already existing in our target. If the function finds an existing item in the target, it will swizzle the appropriate values in the new web map definition. If the function does not find an existing item based on the typeKeyword, it will clone the item. For non-hosted items, it will recreate those items in the resulting web map.

In this case, we'll set it to False since we know we've not cloned this web map before. We want any hosted layers to also clone in our target.

cloned_wm = target.content.clone_items(items=[power_plant_wm_item],
                                      folder=output_folder.name,
                                      search_existing_items=False)

Let's examine the resulting items in the target:

cloned_wm

[<Item title:"US Power Plants" type:Feature Layer Collection owner:api_data_owner>,
 <Item title:"US Power Plant Data" type:Web Map owner:api_data_owner>]

cloned_wm_obj = WebMap(cloned_wm[1])

for wm_lyr in cloned_wm_obj.definition["operationalLayers"]:
    print(f"{wm_lyr['title']}\n{' '*2}{wm_lyr['url']}")
    print(f"{' ' * 2}Host Organization id: {wm_lyr['url'].split('/')[3]}")
    print("\n")

US Power Plants
  https://pythonapi.playground.esri.com/server/rest/services/Hosted/US_Power_Plants/FeatureServer/0
  Host Organization id: server

We can see that clone_items() successfully recreated the web map and cloned the operational layer, and configured the resulting web map to use the cloned feature layer. We can see from the netscheme and path components of the layer URL that the resulting layer is using an ArcGIS Enterprise rather than an ArcGIS Online organization.

Let's visually compare our webmaps:

source

wm_obj

target

cloned_wm_obj

We can see visually that the maps appear to contain the same layers. You can repeat this process for any other web map source/target pair from the wm_map dictionary to make a visual comparison.

ArcGIS Dashboards

See the Cloning and Troubleshooting Complex Items guide for details on cloning ArcGIS Dashboards.

Story Maps

See the Cloning and Troubleshooting Complex Items guide for details on cloning ArcGIS StoryMaps.

Conclusion

This guide demonstrated one workflow for cloning items between a source Web GIS and a target Web GIS. We connected to each GIS as an administrator and cloned an individual item, and then a set of items. We then cloned a Web Map item, demonstrating how a hosted feature layer within the web map is also cloned and the cloned web map is configured to read the cloned feature layer which is hosted in the target organization.