Introduction
A common question the Survey123 team has received from organization administrators is, "What's the best way to clone my surveys from one organization to another?"
There are two common use cases for cloning surveys: 1. Create a copy of a survey in another ArcGIS organization. For example, a city's transportation and water departments have different ArcGIS Online organizations and the water department would benefit from having a copy of one of the transportation department's surveys as well as its associated web map and dashboard. 2. Clone a survey from a development organization in ArcGIS Enterprise to staging and production organizations.
This sample Python notebook demonstrates how to clone surveys and associated content from one organization to another. This workflow can be used to clone surveys from ArcGIS Online to ArcGIS Online, ArcGIS Online to ArcGIS Enterprise, or ArcGIS Enterprise to ArcGIS Enterprise. The direction of cloning does not matter.
This notebook demonstrates two cloning methods:
The foundation of the workflow is the clone_items() method in the ArcGIS API for Python. This is the infrastructure that allows us to clone surveys from a source organization to a target organization. Given the different content and item types, possible ArcGIS Enterprise and ArcGIS Online configurations, security considerations, and item dependencies, the clone_items() method aims to produce an exact duplicate of an item that retains all of its functionality.
Please note that cloning relies on the sharing model to determine the items a user can clone. The user specified in the source organization will need admin access to the content that will be cloned, and the user specified in the target organization will need the ability to create content in that organization.
For more information on the clone_items method, see the ArcGIS API for Python Cloning content guide and API reference.
Prepare to clone
To start, we are going to need two GIS connections: one to our "source" organization, which is the organization in which the survey and content currently resides that we would like to clone; and another to a "target" organization, which is the organization that we would like to clone the survey and content to.
import arcgis
from arcgis.gis import GIS
import os
# Connect to source and target organizations
source = GIS(profile="sourceOrg")
target = GIS(profile="TargetOrg")
print("Source GIS:", source,"\nTarget GIS:", target)Source GIS: GIS @ https://Survey123ninja.maps.arcgis.com version:8.4 Target GIS: GIS @ https://Survey123.maps.arcgis.com version:8.4
The first example highlights a workflow where you have a few surveys shared to a group that you would like to clone to a different organization. In order to work with your surveys a Survey Manager is defined. A survey in the Survey Manager is a single instance of a survey project that contains the item information and properties and provides access to the underlying survey dataset. In this example, four surveys are shared to a group. Using the group ID, a connection is made to the group and a list is created containing all form items in the group.
# Get surveys by group ID and then download each format supported
survey_manager = arcgis.apps.survey123.SurveyManager(source)
group = source.groups.get('0dd43178863545fbb5526278e1bdfc4c')
sourceForms = group.search('type:form')['results']
sourceForms[<Item title:"COVID-19 Community Support" type:Form owner:NinjaGreen>, <Item title:"COVID-19 Inpatient Screening" type:Form owner:NinjaGreen>, <Item title:"COVID-19 Symptoms Check" type:Form owner:NinjaGreen>, <Item title:"COVID-19 Testing Request" type:Form owner:NinjaGreen>]
Now that we have your forms as a list, you are ready to clone the content from the source organization to the target organization. As previously noted, a use case for using the clone_items() method is to clone surveys between development, staging, and production organizations. This first example clones the surveys from an existing group (as defined above) located in the source organization to the target organization and shares the cloned surveys to a group with the same name in the target organization.
Clone related items
The code below starts by creating a new group in the target organization using the same title and tags as the existing group in the source organization. Once the surveys are cloned, they will be shared to this newly created group.
Each form item in the source group is looped through, obtaining the feature service associated with the survey through the Survey2Service relationship, as well as any additional items related to the survey using the Survey2Data relationship. This would include any linked content or report templates associated with the survey. All related items are merged into a list to be cloned. For more information on related items in ArcGIS, see Relationship types.
Next, a new folder in the target organization is created based on the survey name, and the items in the list are cloned. Since the copy_data parameter is set to "False" in the clone_items() method, the resulting services created in the target organization will not contain any data. This is ideal when cloning from a development environment to a staging or production environment, as you might not wish to retain any test data.
Finally, the form items are shared to the group in the target organization that was defined previously, including the feature service, and tags are added to each item.
# Create new group in the target environment
shared_group = target.groups.create(title=group.title, tags=group.tags)
# Clone source items to the target environment
for form_item in sourceForms:
# Obtain the feature service associated with the survey
related_items = form_item.related_items('Survey2Service','forward')
# Obtain the additional items that have a relationship with the survey
additional_items = form_item.related_items('Survey2Data','forward')
all_items = [form_item] + related_items + additional_items
print("Source items: ", *all_items, sep="\n")
# Create new folder according to survey name
title = form_item.title.replace(" ", "_")
folderName = "Survey-" + title
target.content.create_folder(folderName)
# Clone all items to the new folder in target environment
cloned_items = target.content.clone_items(items=all_items, folder=folderName, copy_data=False)
# Check the feature count in cloned feature layer. Feature count should 0 because existing data is not copied
print("Result feature count: ", cloned_items[0].layers[0].query(where='1=1', return_count_only=True))
for item in cloned_items:
if item.type == 'Form':
# Update the form item to ensure all resources are rebuilt
downloaded_item = item.download(file_name=item.id+'.zip')
item.update({}, downloaded_item)
os.remove(downloaded_item)
# Share the form item to the group
item.update(item_properties={'tags':'PrdEnv, PythonAPI'})
item.share(groups=shared_group.id)
# Share source feature service to group
source_fs = item.related_items('Survey2Service','forward')[0]
source_fs.update(item_properties={'tags':'PrdEnv, PythonAPI'})
source_fs.share(groups=shared_group.id)
else:
passSource items: <Item title:"COVID-19 Community Support" type:Form owner:NinjaGreen> <Item title:"COVID-19 Community Support" type:Feature Layer Collection owner:NinjaGreen> <Item title:"COVID-19 Community Support Report Template" type:Microsoft Word owner:NinjaGreen> Result feature count: 0 Source items: <Item title:"COVID-19 Inpatient Screening" type:Form owner:NinjaGreen> <Item title:"COVID-19 Inpatient Screening" type:Feature Layer Collection owner:NinjaGreen> Result feature count: 0 Source items: <Item title:"COVID-19 Symptoms Check" type:Form owner:NinjaGreen> <Item title:"COVID-19 Symptoms Check" type:Feature Layer Collection owner:NinjaGreen> <Item title:"COVID-19 Symptoms Check Report Template" type:Microsoft Word owner:NinjaGreen> Result feature count: 0 Source items: <Item title:"COVID-19 Testing Request" type:Form owner:NinjaGreen> <Item title:"COVID-19 Testing Request" type:Feature Layer Collection owner:NinjaGreen> Result feature count: 0
As a confidence check, let's query the newly created group and confirm our four surveys have been shared.
# Check items in the new group in target environment.
import time
time.sleep(30)
targetForms = target.content.search('type:form group:'+shared_group.id)
print("Cloned surveys: ", *targetForms, sep="\n")Cloned surveys: <Item title:"COVID-19 Symptoms Check" type:Form owner:survey123_publisher> <Item title:"COVID-19 Testing Request" type:Form owner:survey123_publisher> <Item title:"COVID-19 Inpatient Screening" type:Form owner:survey123_publisher> <Item title:"COVID-19 Community Support" type:Form owner:survey123_publisher>
The example above is a useful workflow for cloning surveys between development, staging, and production environments and will only clone items that are linked to the form item, including: report templates, linked web maps, CSVs, and map packages. But what if you would like to clone not only the survey and its related items, but also the survey data that's already been collected? Additionally, what if you have web maps, web apps, or dashboards that use the survey data but aren't linked to the survey?
If those items are stored in the survey's folder, a slightly different method can be used to clone all the content in this folder.
Clone survey folder
The code below demonstrates connecting to one specific survey in your organization. Using the properties of the survey, the folder ID where the survey resides is assigned to a variable. Next, all the folders in the source organization for the source username are listed. Using list comprehension, the folder in the full_folder variable is matched with the folder ID obtained from the survey properties.
Once the correct folder has been identified the contents of the folder are listed.
source_item_with_data = survey_manager.get("65b0ce4cfa2145eb8ce90122e54029e6")
survey_folder = source_item_with_data.properties['ownerFolder']
usr = arcgis.gis.User(source, source.users.me.username)
full_folder = usr.folders
# Identify the folder associated with the survey
fldr = next((f for f in full_folder if f['id'] == survey_folder), 0)
#List all the items within that folder to be cloned later on
fldr_items = usr.items(folder=fldr)
print("Folder items:", *fldr_items, sep="\n")Folder items: <Item title:"Incident Report" type:Form owner:NinjaGreen> <Item title:"Incident Report" type:Feature Layer Collection owner:NinjaGreen> <Item title:"Incident Report_fieldworker" type:Feature Layer Collection owner:NinjaGreen> <Item title:"Incident Report Webmap" type:Web Map owner:NinjaGreen> <Item title:"Incident Report Google Vision" type:Web Map owner:NinjaGreen> <Item title:"Incident Report_sampleTemplate" type:Microsoft Word owner:NinjaGreen> <Item title:"Incident Report Dashboard" type:Dashboard owner:NinjaGreen> <Item title:"Incident Report_stakeholder" type:Feature Layer Collection owner:NinjaGreen> <Item title:"Incident Report_sampleTemplate 2.docx" type:Microsoft Word owner:NinjaGreen>
Now that all items to be cloned are in a list, a new folder is created in the target organization to store the content. After the folder is created the content is cloned to the target environment. Since the copy_data parameter is not defined the default value for the parameter is "True", meaning all the underlying data will also be cloned. This means the resulting content in the target organization will be an identical clone of the original data. If you do not wish to retain the source data, setting the copy_data parameter to "False" will only clone the data schema and architecture to the target organization. The survey, web maps, web apps, dashboards, and other items will be configured as per their original items; the only difference is the feature layer will be empty.
# Create a new folder with the same name as the source folder to clone the contents to
target.content.create_folder(folder=fldr['title']+"_Python")
# Clone items to the new folder
cloned_items = target.content.clone_items(items=fldr_items, folder=fldr['title']+"_Python")
print(*cloned_items, sep="\n")
print("Result feature count: ", cloned_items[0].layers[0].query(where='1=1', return_count_only=True))
# Search for the cloned survey and update the form item to ensure all resources are rebuilt
search_clone_survey = target.content.search(f"title: {source_item_with_data.properties['title']} AND owner: {target.users.me.username} Form")
cloned_survey = search_clone_survey[0]
download_survey = cloned_survey.download(file_name=cloned_survey.id+'.zip')
cloned_survey.update({},download_survey)
os.remove(download_survey)<Item title:"Incident Report" type:Feature Layer Collection owner:survey123_publisher> <Item title:"Incident Report_sampleTemplate" type:Microsoft Word owner:survey123_publisher> <Item title:"Incident Report_sampleTemplate 2.docx" type:Microsoft Word owner:survey123_publisher> <Item title:"Incident Report_fieldworker" type:Feature Layer Collection owner:survey123_publisher> <Item title:"Incident Report Google Vision" type:Web Map owner:survey123_publisher> <Item title:"Incident Report_stakeholder" type:Feature Layer Collection owner:survey123_publisher> <Item title:"Incident Report" type:Form owner:survey123_publisher> <Item title:"Incident Report Webmap" type:Web Map owner:survey123_publisher> <Item title:"Incident Report Dashboard" type:Dashboard owner:survey123_publisher> Result feature count: 111
This notebook covered two use cases for the clone_items() method and is intended to be used as a guide; you can take what's here and incorporate it into your own workflows.
What workflows or use cases do you have that we missed? Please let us know your use cases and workflows and we can work on incorporating them into the notebook.
Notes on limitations:
- Clone fails with non-ASCII characters in service name.
- Cloning is limited to 1000 records.
- BUG-000136846 - The clone_items() method fails when attempting to clone a public hosted feature layer view hosted by another organization with the error message, "User does not have permissions to access this service."
- BUG-000141004 - ArcGIS API for Python clone_items() method isn’t re-creating the item info URL’s for surveys published from the web designer.
- The workaround is to download the survey from the target environment and immediatly update it using the file downloaded.