Update attachment keywords

This notebook uses the ArcGIS API for Python. For more information, see the ArcGIS API for Python documentation and guides.

ArcGIS Survey123 uses attachment keywords to associate an attachment in a feature layer with its corresponding image, file, or audio question in a survey. Attachment keywords are supported for hosted feature layers in ArcGIS Online and ArcGIS Enterprise version 10.8.1 and later.

There are three main reasons you would need to update the keywords for your feature service attachments:

You've exported your Survey123 data as a file geodatabase and want to use that data in Survey123 again
You've authored a feature class, enabled attachments, published as an ArcGIS Server feature service, and started collecting data in Survey123
You've upgraded your ArcGIS Enterprise organization from a pre-10.8.1 version to a post 10.8.1 version and want to update your existing data to now make use of keywords

Python notebook to update attachment keywords for existing attachments.

The workflow assumes all attachments were captured or uploaded in the Survey123 field app or captured in the Survey123 web app (but not uploaded in the web app).

Please refer to the Keywords caution note in the Media questions Survey123 documentation to determine if keywords are enabled on your feature service.

import logging
import os
import re
import datetime
import requests
import tempfile
from IPython.display import display
from arcgis.gis import GIS

Start by defining your variables. The variables are as follows:

feature_layer_id - Item ID for the feature service associated with the survey.
portal_username - Organization username.
portal_password - Organization password.
portal_url - The URL for your organization (e.g. www.arcgis.com for ArcGIS Online).
multiple_image_questions - Does your survey have multiple image questions? Accepts yes or no. yes means that your survey contains multiple image questions; no means that your survey contains only one image question (this question can use use multiline appearance).
attachment_keyword - Only relevant if there is one image question in your survey. The script will prompt for the keyword, otherwise the script will extract the attachment keyword from the photo name if there are multiple image questions. For more information on how to identify the correct attachment keyword(s) for your survey, see the How To: Update the attachment keywords for existing ArcGIS Survey123 data Knowledge Base article.

# What is the ID of the hosted feature layer associated with your survey?
feature_layer_id = '<itemID>'
portal_username = '<OrgUsername>'
portal_password = '<OrgPassword>'
portal_url = 'https://<FQDN>/<PortalWebAdaptor>'

# Do you have multiple image questions in your survey?
# 'yes' means you do have multiple image questions in your survey
# 'no' means you only have one image question (can use multiline appearance)
multiple_image_questions = 'yes'

# If one image question, obtain attachment keyword from user else grab it from the attachment name later on
if multiple_image_questions == "no":
    attachment_keyword = str(input("Please enter the attachment keyword to use: "))
else:
    attachment_keyword= ''

The next steps are to connect to your GIS, obtain the token for your session, and make a connection to the feature layer.

# Connect to GIS and get feature layer information
if portal_username == '' and portal_password == '':
    gis = GIS(profile='Survey123_prof')
else:
    gis = GIS(portal_url, portal_username, portal_password)

token = gis._con.token
item_object = gis.content.get(feature_layer_id)

The update_attachment() function is used to update the keyword for each attachment. The arguments for the function are as follows:

REST URL of the feature layer
Token generated earlier
Current object ID
File path to local copy of the attachment
Attachment ID
Keyword to apply to each attachment

Once all arguments are obtained the function constructs the URL using the REST endpoint and current object ID. It then opens the attachment to be used as a file in the POST request. Next, it defines the request parameters with the remaining input arguments and sends the POST request.

The JSON response indicates if the request was successful or not.

def update_attachment(url, token, oid, attachment, attachID, keyword):
    att_url = '{}/{}/updateAttachment'.format(url, oid)
    start, extension = os.path.splitext(attachment)

    jpg_list = ['.jpg', '.jpeg']
    png_list = ['.png']
    if extension in jpg_list:
        files = {'attachment': (os.path.basename(attachment), open(attachment, 'rb'), 'image/jpeg')}
    elif extension in png_list:
        files = {'attachment': (os.path.basename(attachment), open(attachment, 'rb'), 'image/png')}
    else:
        files = {'attachment': (os.path.basename(attachment), open(attachment, 'rb'), 'application/octect-stream')}

    params = {'token': token,'f': 'json', 'attachmentId': attachID, 'keywords': keyword}
    r = requests.post(att_url, params, files=files)
    return r.json()

If there are multiple image questions in the survey, the findattachmentkeyword() function below is used to obtain the keyword from the name of each attachment. The function takes the attachment name as the input and extracts the text before the hyphen ('-') or underscore ('_').

By default, attachments captured or uploaded in the Survey123 field app are named using the format <attachmentKeyword>-<timestamp>, and attachments captured in the Survey123 web app are named using the format <attachmentKeyword>_<timestamp>. The function only accepts attachment names in these two formats. All other names are ignored. There are two cases where the attachment name might not be in an acceptable format:

The name was modified from the default by the user.
The attachment was uploaded in the web app (as opposed to it being captured in the form by camera, signature, and so on). When an attachment is uploaded in the web app it retains the name of the source file.

def findattachmentkeyword(attach_name):
    kw = ''
    # For attachments submitted in the field app
    if any("-" in s for s in attach_name):
        part = attach_name.partition("-")
        kw = part[0]
    # For attachments submitted in the web app
    elif any("_" in s for s in attach_name):
        part = attach_name.partition("_")
        kw = part[0]

    return kw

The function below runs the entire workflow.

First, the function creates a temporary directory to save all the attachments to.
It then proceeds with attachments for all layers and tables in the feature service that have attachments enabled.
For each layer, the function queries the features in the layer and obtains a list of attachments for each object ID.
Each attachment is then downloaded to the temporary directory and is then updated using the update_attachment() function, including the attachment keyword either entered above or obtained from the findattachmentkeyword() function.

def update_attachments():
    with tempfile.TemporaryDirectory() as tmp:
        tmp_path = tmp
        layers = item_object.layers + item_object.tables
                    
        for layer in layers: 
            url = layer.url
            # Skip layer if attachments are not enabled
            if layer.properties.hasAttachments == True:
                # Remove any characters from feature layer name that may cause problems and ensure it's unique
                feature_layer_name = '{}-{}'.format(str(layer.properties['id']), re.sub(r'[^A-Za-z0-9]+', '', layer.properties.name))
                feature_layer_folder = tmp_path + feature_layer_name

                # Query to get list of object IDs in layer
                feature_object_ids = layer.query(where='1=1', return_ids_only=True)
                for j in range(len(feature_object_ids['objectIds'])):
                    current_oid = feature_object_ids['objectIds'][j]
                    current_oid_attachments = layer.attachments.get_list(oid=current_oid)

                    if len(current_oid_attachments) > 0:
                        for k in range(len(current_oid_attachments)):
                            attachment_id = current_oid_attachments[k]['id']
                            attachment_name = current_oid_attachments[k]['name']
                            current_folder = os.path.join(feature_layer_folder, str(current_oid))
                            file_name = '{}-{}'.format(attachment_id, attachment_name)
                            current_attachment_path = layer.attachments.download(oid=current_oid,
                                                                                      attachment_id=attachment_id,
                                                                                      save_path=current_folder)
                            if len(attachment_keyword) > 0:
                                request = update_attachment(url, token, current_oid, current_attachment_path[0]
                                                            , attachment_id, attachment_keyword)
                                print("Completed updating attachment on feature layer", feature_layer_name,"with ID", str(attachment_id), "on ObjectID", str(current_oid), "\n", "With the response of", request)
                            else:
                                found_kw = findattachmentkeyword(attachment_name)
                                request = update_attachment(url, token, current_oid, current_attachment_path[0]
                                                        , attachment_id, found_kw)
                                print("Completed updating attachment on feature layer", feature_layer_name,"with ID", str(attachment_id), "on ObjectID", str(current_oid), "\n", "With the response of", request)

                            os.remove(current_attachment_path[0])
                            os.rmdir(current_folder)
                os.rmdir(feature_layer_folder)
                            

update = update_attachments()

Completed updating attachment on feature layer 0-AttachmentManholeInspectionMultipleLayers with ID 1 on ObjectID 1 
 With the response of {'updateAttachmentResult': {'objectId': 1, 'globalId': '{1804AA9F-4270-4575-A006-2A3B3CA2A370}', 'success': True}}
Completed updating attachment on feature layer 0-AttachmentManholeInspectionMultipleLayers with ID 2 on ObjectID 2 
 With the response of {'updateAttachmentResult': {'objectId': 2, 'globalId': '{331182FF-25AC-4B90-9998-632F2DB1EAB8}', 'success': True}}
Completed updating attachment on feature layer 0-AttachmentManholeInspectionMultipleLayers with ID 3 on ObjectID 3 
 With the response of {'updateAttachmentResult': {'objectId': 3, 'globalId': '{84C43A0A-F2B8-4AE5-8A07-1A439DEA6DDF}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 1 on ObjectID 1 
 With the response of {'updateAttachmentResult': {'objectId': 1, 'globalId': '{D60DAFD3-B01C-4065-9397-55EBB1137FF9}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 2 on ObjectID 1 
 With the response of {'updateAttachmentResult': {'objectId': 2, 'globalId': '{3C94303D-E26F-40BB-A58C-B5A60E460735}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 3 on ObjectID 1 
 With the response of {'updateAttachmentResult': {'objectId': 3, 'globalId': '{55FC99FB-86A0-460D-9EA7-904DF1F8CD29}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 4 on ObjectID 1 
 With the response of {'updateAttachmentResult': {'objectId': 4, 'globalId': '{53B288BA-78A0-4785-8666-CA7AA9E9BE2C}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 5 on ObjectID 2 
 With the response of {'updateAttachmentResult': {'objectId': 5, 'globalId': '{1E2901D2-8F29-4B33-9664-0791CBE584CC}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 6 on ObjectID 2 
 With the response of {'updateAttachmentResult': {'objectId': 6, 'globalId': '{3DF38311-0C43-465F-925F-B694A4A0DC4A}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 7 on ObjectID 2 
 With the response of {'updateAttachmentResult': {'objectId': 7, 'globalId': '{27944AE4-2FEC-4B96-A2E9-87171A4F06B3}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 8 on ObjectID 2 
 With the response of {'updateAttachmentResult': {'objectId': 8, 'globalId': '{4B004255-7CEF-45CD-918F-0AC50B77446A}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 9 on ObjectID 3 
 With the response of {'updateAttachmentResult': {'objectId': 9, 'globalId': '{D451D3C9-03A4-4238-8A73-E93197C40A84}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 10 on ObjectID 3 
 With the response of {'updateAttachmentResult': {'objectId': 10, 'globalId': '{312D5E72-6E3D-47CD-93C9-43C2E05E0328}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 11 on ObjectID 3 
 With the response of {'updateAttachmentResult': {'objectId': 11, 'globalId': '{BC7CA91F-EB10-4A5E-9D99-7AAC77223FF5}', 'success': True}}
Completed updating attachment on feature layer 0-defects with ID 12 on ObjectID 3 
 With the response of {'updateAttachmentResult': {'objectId': 12, 'globalId': '{43DD83EE-C399-4A83-9F1A-29606F73655A}', 'success': True}}