Bike Paths in East Central MN

Image of person riding a bicycle

Arcpy generated Map output from programmatically accessed polygon and line feature class data

Multiple Shapefile data sources accessed programmatically online and loaded into a geodatabase in Arcpy,
The city/township polygon and bike path line feature data were loaded onto an existing project map aprx file
Result for east central Minnesota shows predominant bike path types and total route lengths
Off road paths are predominant in east central Minnesota in areas surrounding Minneapolis and St. Paul
- Unknown indicates that the predominant route type was populated with a value of unknown

What is Arcpy?:¶

Reminder: Arcpy is a Python library for automating GIS tasks in Arc GIS Pro

Collection of modules, functions, and classes from the ArcGIS toolbox
It allows you to perform geoprocessing, analysis, data management and mapping automation using python

In [1]:

import arcpy # allows for access to ArcGIS pro geoprocessing tools and workflow automation
import os # enables interaction with local system resources (paths to folders and files)
import requests # access data from the web
import zipfile ## need this to process and extract downloaded zipfiles later on
import pandas as pd ## use to check spreadsheet formatting


initial_dir= os.getcwd() # capture the initial directory before setting environment for arcpy
print("Initial working directory: ", initial_dir)

Initial working directory:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS

In [2]:

### BE SURE TO INSTALL MAGIC LIBRARY FIRST for filetype validation example below : 
# step1 activate arcgispro-py3 environment: conda activate "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3"
# step2 install library: pip install python-magic-bin on windows, for unix system use conda install -c conda-forge python-magic

import magic ## to validate file type requested from web

What we will do this time:¶

Use custom functions to list feature classes, download and load shapefile programmatically, etc
Create new functions (e.g.,using the request library to pull shape file from MN GeoCommons)
Automate loading of new data onto a map of an existing ArcGIS project

Create and re-use Custom Functions to Save time¶

When you create a python script, the functions within those scripts can be loaded into other scripts
To import your script, do as you would any library in python
if your script is called my_script.py, then:
- python import my_script as ms
Now the functions within this script are available to use
- python ms.my_function()
If you would like these tools to test or follow along download the script file here: Download

In [4]:

## We have such a script to use
# So, let's load in our custom tool
import custom_arcpy_tools as cat

Initial working directory:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS

In [5]:

cat.listFC_dataset_Attributes()

## We purposefully caused an error
# The error is due to workspace not being set in this new notebook or script

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_55452\4143424007.py in <cell line: 0>()
----> 1 cat.listFC_dataset_Attributes()
      2 
      3 ## We purposefully caused an error
      4 # The error is due to workspace not being set in this new notebook or script

c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\custom_arcpy_tools.py in listFC_dataset_Attributes(workspace)
     92     if not arcpy.env.workspace:
     93 
---> 94         raise ValueError("No workspace is set! Please specify a workspace") #stops the program, calls out this error
     95 
     96 

ValueError: No workspace is set! Please specify a workspace

In [6]:

### Don't remember the arguments for this function? 
### Use function_name??

In [7]:

help(cat.listFC_dataset_Attributes) ## use help(function_name) to get the documentation for the function

Help on function listFC_dataset_Attributes in module custom_arcpy_tools:

listFC_dataset_Attributes(workspace=None)
    List feature classes in the workspace and checks their key attributes, such as SpatialReference , 
    geometry type, and coordinate systems
    Parameters:
    -----------
    workspace: str, optional
        The file path to the workspace (e.g., geodatabase path) containing the feature classes
        If none, the function will use the current ArcPy workspace
        If no workspace is set will raise an error telling you to set a a workspace
    
    Returns:
    --------
    None , if no workspace exists
    Otherwise,    Prints the details of all feature classes in the wksp , including
                - Feature Class Name
                - spatial reference name
                - spatial reference type
                - Geometry
                - Well-known ID of the spatial reference
                Also, warns if different spatial references or coordinate systems exists in the dataset
    Raises: 
    -------
    Value Error: 
        If no wksp is set or if not feature classes are found in the workspace
    Exception:
        For general errors that may occur during processing
    
    Example:
    # To list fc in a geodatabase
    >>> listFC_dataset_Attributes(r"c:/Path/To/Your/Workspace.gdb")
    
    # To use the current wksp already set in Arcpy
    >>> listFC_dataset_Attributes()

In [8]:

## Can use magic commmands in the notebook to get file paths
## 
%lsmagic ## lists out all magic commands

Out[8]:

Available line magics:
%alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cd  %clear  %cls  %colors  %conda  %config  %connect_info  %copy  %ddir  %debug  %dhist  %dirs  %doctest_mode  %echo  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %macro  %magic  %matplotlib  %mkdir  %more  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %ren  %rep  %rerun  %reset  %reset_selective  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%cmd  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

In [9]:

## will use %pwd 
# to check our working directory and look for the geodatabase name
%pwd

Out[9]:

'c:\\Projects\\my_git_pages_website\\Py-and-Sky-Labs\\content\\GIS'

In [10]:

# list the files in the current folder
%ls

 Volume in drive C is OS
 Volume Serial Number is DA19-2472

 Directory of c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS

04/06/2025  10:02 AM    <DIR>          .
04/02/2025  09:38 AM    <DIR>          ..
04/06/2025  10:02 AM    <DIR>          __pycache__
04/06/2025  10:01 AM            55,590 BikePaths.ipynb
02/19/2025  12:25 PM                78 category.md
04/02/2025  09:48 AM            23,301 custom_arcpy_tools.py
03/20/2025  10:51 PM            38,607 Disaster Mapping.md
02/19/2025  11:20 AM            12,312 folium_map.ipynb
02/19/2025  11:21 AM             9,048 folium_map2.ipynb
03/06/2025  03:35 PM    <DIR>          InputData
02/26/2025  12:02 AM               567 resize_pngImage.py
03/04/2025  04:18 PM    <DIR>          Tools
               7 File(s)        139,503 bytes
               5 Dir(s)  418,605,436,928 bytes free

In [ ]:

## Not here
## Change the directory to the folder with the gdb

In [11]:

%cd ./02 Result

[WinError 2] The system cannot find the file specified: './02 Result'
c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS

In [ ]:

## Folders for the Processed Results and Output Map doesn't exist, make some folders

In [ ]:

%mkdir "./02 Result" "./03 Map"

In [16]:

# list the files again
%ls

 Volume in drive C is OS
 Volume Serial Number is DA19-2472

 Directory of c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS

04/06/2025  10:08 AM    <DIR>          .
04/02/2025  09:38 AM    <DIR>          ..
04/06/2025  10:02 AM    <DIR>          __pycache__
04/06/2025  10:05 AM    <DIR>          02 Result
04/06/2025  10:05 AM    <DIR>          03 Map
04/06/2025  10:05 AM            55,718 BikePaths.ipynb
02/19/2025  12:25 PM                78 category.md
04/02/2025  09:48 AM            23,301 custom_arcpy_tools.py
03/20/2025  10:51 PM            38,607 Disaster Mapping.md
02/19/2025  11:20 AM            12,312 folium_map.ipynb
02/19/2025  11:21 AM             9,048 folium_map2.ipynb
03/06/2025  03:35 PM    <DIR>          InputData
02/26/2025  12:02 AM               567 resize_pngImage.py
03/04/2025  04:18 PM    <DIR>          Tools
               7 File(s)        139,631 bytes
               7 Dir(s)  418,627,325,952 bytes free

In [17]:

## custom function to create sub-folder within the folder where your script exists
### Project Folder/
##  |---Script.py
##      |---Subfolder

## check where the script is and make a folder
def get_path_mkfolder(make_folder=False,folder_name=None):
    """Get the path to your script or notebook and optionally creates a folder in at the same level
        rootFolder/
        |--ScriptFolder/
            |---Script.py
            |---New Folder

    Args:
        make_folder (bool, optional): Set to True to make a new folder. Defaults to False.
        folder_name (str, optional): Name of the Folder to be created. Defaults to None.
    Return:
        str: The path of the created folder, or the script directory
    """
    try:
        script_path=os.path.dirname(os.path.abspath(__file__))
        print("Running a python script file here: ", script_path)

        if make_folder and folder_name:
            subfolder_path = os.path.join(script_path,folder_name)

            #check if the folder exists 
            if not os.path.exist(subfolder_path):
                os.makedirs(subfolder_path)# make the folder
                print("Created subfolder at: ", subfolder_path)
            else:
                print("Folder already exists: ",subfolder_path)
            return subfolder_path # return subfolder path

        print("Script in this folder: ", script_path)
        return  script_path # return script path
    
    # handle typos, undefined paths
    except NameError:
        # handle cases where __file__ is not available , user is in a notebook
        script_path = initial_dir
        print("Running in a notebook here: ", script_path)
        if make_folder and folder_name:
            subfolder_path = os.path.join(script_path,folder_name)
            # check if folder exists
            if not os.path.exists(subfolder_path):
                os.makedirs(subfolder_path) # create the folder
                print("Created subfolder at: ", subfolder_path)
            else:
                print("Folder already exists: ", subfolder_path)
            return subfolder_path
        
        print("Script in this folder: ", script_path)
        return script_path

In [18]:

folder_path =get_path_mkfolder(make_folder=True, folder_name="01 Data") # if following along, for consistency with later parts, change to "01 Data"

print("Folder path to use", folder_path)

Running in a notebook here:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS
Created subfolder at:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data
Folder path to use c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data

In [ ]:

## Now we can use this folder to store new inputs, and manage the folder organization
## Here the project already had a defined folder layout, and we could have re-used the 01 Data folder to store new inputs
## rootFolder:
#   |---Script.ipynb   
#   |---01 Data Folder
#       |--- .shp files  
#   |---02 Results Folder
#       |--- .gdb
#   |--03 Map
#   |--04 TestFolder

In [20]:

# prefixing with ! and using cd.. & tree will run system command tree to get a tree like view of the project folder
! tree

Folder PATH listing for volume OS
Volume serial number is DA19-2472
C:.
+---01 Data
+---02 Result
+---03 Map
+---InputData
+---Tools
+---__pycache__

Data Sources :¶

Geography: MN

Input Name	file types	Coordinate System	Source
Bikeways in Minnesota	1 shapefile	PCS: NAD 83 UTM Zone 15N	MetaData
MN Cities and townships	1 shapefile	PCS: NAD 83 UTM Zone 15N	MetaData

Now will retrieve the bikepaths and cities and townships data

Get online data with the Requests library¶

`requests.get(url, params= None, **kwargs)`¶

Required argument : url (url to send the request to)
params(optional, default is None), allows you to pass in query parameters that filter, modify or sort the data you're requesting
- response = response.get("https://some.example.com/data", params ={key:value})
**kwargs: keyword arguments. A way to pass additional arguments to the request function as a dictionary
- can pass optional keyword arguments, headers, timeout
Use-case: make customized requests to web pages, API or get files
- Example request with parameters:
- response = response.get("https://some.example.com/data", params={"State":"Minnesota"}, headers="User-Agent":"my-app")

In [23]:

###===================   Automate ShapeFile dataset download: access, download and store new input shapefile dataset from a url 

# Ofcourse, we can always navigate to MN Geocommons or elsewhere, click, download and place the zipfile into our data input folder
# Would have to do this every time, i.e., creating folders, navigating to extracted folder, extracting files
# The downloadShapefile Function below accomplishes all this automatically using only the target url, and optional target folder
    # Note: it re-uses are previous function to get the script path and use that to create the target folder for storing data


def downloadShapefile(url, target_folder=None):
    """Access and download a shapefile from a url pointing to the shapefile

    Args:
        url (str): Url of the Shapefile. Uses the  requests library to access the file header 
            handles large file size by not streaming all content into memory if the file is above 200 megabytes
            downloads the zip file and then extracts to a local path
        target_folder(str, optional): path to the folder where files should be stored. 
            target_folder defaults to "01 Data" which is created in the same folder where the script exists
    Returns:
        Tuple of File paths, str:  Returns the path to the downloaded zip file
    """
    try:
        # Set up folders where data will be stored
        data_folder = target_folder if target_folder else get_path_mkfolder(True, "01 Data")
        print("Subfolder will be created within: ", data_folder)

        # Create folders for the zipfile download and to hold the extracted files
        file_path = os.path.join(data_folder, r"ShapeFile_Inputs\downloadedData.zip")
        extracted_zipFolder = os.path.join(data_folder, r"ShapeFile_Inputs\ExtractedZip") 
        
        # Check folders exist
        if not os.path.exists(os.path.dirname(file_path)):
            os.makedirs(os.path.dirname(file_path)) # make the folder to hold the download if it does not exists
            print("Set the path for the download to: ", file_path)
        if not os.path.exists(extracted_zipFolder):
            os.makedirs(extracted_zipFolder) # make the folder to hold the extracted files if it does not exists
            print("Set the path for Extracted zip files to", extracted_zipFolder)
        else:
            print("Extracted Zip folder exists, please validate the output exists: ", extracted_zipFolder)
                
            # get the file size
            response_head= requests.head(url)

            if 'Content-Length' in response_head.headers:
                response_header= response_head.headers

                file_size = response_header['Content-Length']

                file_size_mb= round(int(file_size) / (1000000)) # convert bytes to megabytes
                print(f"Shapefile Data File size ~: {file_size_mb} megabytes")
            else:
                print("Content length header not found in response. Proceeding with download")

            # check file size
            if file_size_mb <200:
                # Download file
                response= requests.get(url)
                with open(file_path, "wb") as f:
                    f.write(response.content)

            else: # Download file in chunks
                response = requests.get(url, stream=True)
                with open(file_path, "wb") as f:
                    for chunk in response.iter_content(chunk_size=65536): # chunk size to load into memory from the response
                        if chunk:
                            chunk.write(response.content)

            print("Download Successful")

            ## Validate the filetype is zip
            if magic.from_file(file_path, mime=True) == "application/zip": # extracts, checks the filetype
                print("Extracting data from zip file...")
                # read the zip file
                with zipfile.ZipFile(file_path, "r") as zip_f:
                    zip_f.extractall(extracted_zipFolder) # extract the zipfile to the target folder
                    print("Extraction complete. Files extracted to", extracted_zipFolder)

            else:
                print("Error : Downloaded file is not a valid zip file")
                return None, None
            
            return extracted_zipFolder, file_path # note it returns a tuple that needs to be unpacked for the first element [0]
        
    # handle url request errors
    except requests.exceptions.RequestException as e:
        print(f"Error occurred during the Download{e}")
        return None
    # handle general errors
    except Exception as e:
        print("Unexpected error", e, type(e).__name__)
        return None, None

In [24]:

urls =["https://resources.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_dot/bdry_mn_city_township_unorg/shp_bdry_mn_city_township_unorg.zip","https://resources.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_dot/trans_bikeway/shp_trans_bikeway.zip"]

# returns a tuple of paths that are useful for easily accessing the extracted shapefiles and zipfile download itself
try:
    for url in urls:
        extracted_zipFolder = downloadShapefile(url)[0]
        # take the first item in the tuple to store the location of the shapefiles
except Exception as e:
    print("Error occured", e, type(e).__name__)

Running in a notebook here:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS
Folder already exists:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data
Subfolder will be created within:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data
Extracted Zip folder exists, please validate the output exists:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data\ShapeFile_Inputs\ExtractedZip
Shapefile Data File size ~: 5 megabytes
Download Successful
Extracting data from zip file...
Extraction complete. Files extracted to c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data\ShapeFile_Inputs\ExtractedZip
Running in a notebook here:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS
Folder already exists:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data
Subfolder will be created within:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data
Extracted Zip folder exists, please validate the output exists:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data\ShapeFile_Inputs\ExtractedZip
Shapefile Data File size ~: 16 megabytes
Download Successful
Extracting data from zip file...
Extraction complete. Files extracted to c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data\ShapeFile_Inputs\ExtractedZip

In [25]:

print(extracted_zipFolder)

c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\01 Data\ShapeFile_Inputs\ExtractedZip

In [ ]:

# We now have the dataset stored locally, if done regularly could modify this download to occur on a regular schedule 
    # example: download shapefile dataset from a url weekly, daily

In [29]:

### Change the folder to store the result
%cd "./02 Result"

c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result

In [ ]:

### Use the path of the Result folder and Create new GDB

outfolder= r"c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result"
gdb_name= "savageMN_BikePaths.gdb"
gdb_path= os.path.join(outfolder, gdb_name)

if not os.path.exists(gdb_path):
    arcpy.management.CreateFileGDB(outfolder,gdb_name)
    print("Successfully created Geodatabase")

else: 
    print("Geodatabase already exists")

Successfully created Geodatabase

In [ ]:

gdb_path= os.path.join(outfolder, gdb_name)

print("Geodatabase Full path is here: ", gdb_path)

Geodatabase Full path is here:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result\savageMN_BikePaths.gdb

In [ ]:

### Set workspace to geodatabase 
arcpy.env.workspace = gdb_path
wksp = gdb_path

arcpy.env.overwriteOutput= True

In [ ]:

# quick reminder of where our current workspace is set
print("GeoDatabase is here: " , wksp)

GeoDatabase is here:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result\savageMN_BikePaths.gdb

Bulk importing shapefiles into a Geodatabase¶

`FeatureClassToGeodatabase()`¶

arcpy.conversion.FeatureClassToGeodatabase(Input_Features, Output_Geodatabase)
can take a single input or a list of inputs

In [36]:

## Search the documentation for arcpy.conversion.FeatureClassToGeodatabase()

from IPython.display import IFrame # a module for controlling notebook outputs, allows you to embed images, video, webpages with the Iframe function

# ArcGIS Pro documentation URL for a specific tool
tool_url = "https://pro.arcgis.com/en/pro-app/latest/tool-reference/conversion/feature-class-to-geodatabase.htm"

# Display the documentation inside Jupyter Notebook
IFrame(tool_url, width="100%", height="600px") # iframe can be used to display local or online webpages, documents, reports, visualizations , videos

Out[36]:

In [42]:

###==========================    Load all the shapefiles into a GDB



def loadShapefilesToGDB(gdb_path, shapefile_inputs):
    """Loads all shapefiles in a folder into a geodatabase

    Args:
        gdb_path (str): Path to the geodatabase
        shapefile_inputs (str): Path to the folder holding the shapefiles

    Returns:
        list: Returns  a list of shapefile paths that were used for loading into the GDB, or an emtpy list if no shapefiles were processed
    """
    try:
        # use extracted zipfiles if the path exists, otherise use the specified path to the files
        if not shapefile_inputs or not os.path.exists(shapefile_inputs):
            print("Invalid path to shapefile folder provided")
            return [] # return empty list
        
        # Loop through the folder with extracted files and compile a list of all the shape file paths
        shapefile_list = [] # start an empty list to hold .shp files

        for f in os.listdir(shapefile_inputs):
            if f.endswith(".shp"):
                full_shp_path= os.path.join(shapefile_inputs,f) # get the full path for each file
                shapefile_list.append(full_shp_path) # add to a list

        ###================================  Batch Convert shapefile to Feature classes in the Gdb
        # check the list of shp files exists
        if not shapefile_list:
            print("no shape files found for extraction")
            return []
        
        else:
        ## Load the shapefiles into the gdb
            ### TOOL: arcpy.conversion.FeatureClassToGeodatabase(Input_Features, Output_Geodatabase)
            arcpy.conversion.FeatureClassToGeodatabase(shapefile_list,gdb_path)
            print(f" Successfully Loaded shapefiles into into the {gdb_path}")
            
            return shapefile_list
    
    except Exception as e:
        print("Error Loading Shapefiles into the GDB ", e, type(e).__name__)
    except arcpy.ExecuteError:
        print("Arcpy Error: ", arcpy.GetMessages(2))

In [43]:

# we do not need the input shapefile paths, but for demonstration the list is extracted
loaded_shapefiles=loadShapefilesToGDB(wksp,extracted_zipFolder)

print("These files were loaded into the GDB", loaded_shapefiles)

 Successfully Loaded shapefiles into into the c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result\savageMN_BikePaths.gdb
These files were loaded into the GDB ['c:\\Projects\\my_git_pages_website\\Py-and-Sky-Labs\\content\\GIS\\01 Data\\ShapeFile_Inputs\\ExtractedZip\\Bikeways_in_Minnesota.shp', 'c:\\Projects\\my_git_pages_website\\Py-and-Sky-Labs\\content\\GIS\\01 Data\\ShapeFile_Inputs\\ExtractedZip\\city_township_unorg.shp']

In [46]:

## Check the gdb 

arcpy.ListFeatureClasses()

Out[46]:

['Bikeways_in_Minnesota', 'city_township_unorg']

In [ ]:

## We successfully programmatically retrieved datasets from MN GeoCommons and loaded them into a Geodatabase in ArcGIS Pro

In [ ]:

##  Use the Describe Object to access properties of a Feature class like its Spatial Reference

## Get all the fc classes from the gdb
list_fc = arcpy.ListFeatureClasses()


for fc in list_fc:
    desc= arcpy.Describe(fc)
    sr = desc.SpatialReference

    print(f"Feature class: , {fc}, Spatial Reference name :, {sr.name}, Spatial Ref Type : {sr.type}, Geometry{desc.shapetype}, WKID: {sr.factorycode}")
    print("-"*150,"\n") # add line and space between prints

Feature class: , Bikeways_in_Minnesota, Spatial Reference name :, NAD_1983_UTM_Zone_15N, Spatial Ref Type : Projected, GeometryPolyline, WKID: 26915
------------------------------------------------------------------------------------------------------------------------------------------------------ 

Feature class: , city_township_unorg, Spatial Reference name :, NAD_1983_UTM_Zone_15N, Spatial Ref Type : Projected, GeometryPolygon, WKID: 26915
------------------------------------------------------------------------------------------------------------------------------------------------------

In [48]:

# We already have a custom function that gathers key feature class attributes and prints the workspace to avoid repetitive coding
cat.listFC_dataset_Attributes()

Workspace is set here c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result\savageMN_BikePaths.gdb

Feature class: , Bikeways_in_Minnesota, Spatial Reference name :, NAD_1983_UTM_Zone_15N, Spatial Ref Type : Projected, GeometryPolyline, WKID: 26915
------------------------------------------------------------------------------------------------------------------------------------------------------ 

Feature class: , city_township_unorg, Spatial Reference name :, NAD_1983_UTM_Zone_15N, Spatial Ref Type : Projected, GeometryPolygon, WKID: 26915
------------------------------------------------------------------------------------------------------------------------------------------------------ 


2 Feature classes found in the workspace path

Accessing ArcGIS Pro Projects¶

We can automate the modification and display of data within existing project files using arcpy.mp module
Use arcpy.mp to modify objects in existing projects (.aprx) or layer files (.lyr or .lyrx)
Example use cases:
- Replace a layer's data source
- Iterate through a series of extents, find and replace text values, and export the page layout to PDF
- Build a complete map book by appending PDF documents into a final product

`arcpy.mp.ArcGISProject()`¶

Provides a reference to an ArcGIS project (.aprx) stored on disk or to the project currently loaded in ArcGIS Pro
Two input arguments : path or CURRENT
- path: full path the project file
- CURRENT: keyword that sets the project to the one currently loaded in ArcGIS Pro
  - Must be run from within ArcGIS Pro(e.g., python window or a geoprocessing script tool)

In [ ]:

## check the current path

In [49]:

%pwd

Out[49]:

'c:\\Projects\\my_git_pages_website\\Py-and-Sky-Labs\\content\\GIS\\02 Result'

In [ ]:

## change the directory

In [62]:

%cd ..

c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS

In [ ]:

## Set the directory to the 03 Map folder (where the .aprx ArcGIS Project exists)

In [64]:

%cd "./03 Map"

c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\03 Map

In [ ]:

### Set the path to the folder and file holding the map 
map_folder= r"c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\03 Map"
aprx_name = "TemplateProject.aprx"
aprx_path = os.path.join(map_folder,aprx_name)

if not os.path.exists(aprx_path):
    print("Project path does not exist", os.path.exists(aprx_path))

# Output .aprx with new name to avoid locks or read issues
new_aprx_name= "savage_bikePaths_edit.aprx"
new_aprx_path = os.path.join(map_folder,new_aprx_name) 


## Set gdb path
outfolder= r"c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result"
gdb_name= "savageMN_BikePaths.gdb"
gdb_path= os.path.join(outfolder, gdb_name)



# Load the ArcGIS Pro project object to give us access to the contents of that project
aprx =  arcpy.mp.ArcGISProject(aprx_path)
print(aprx.isReadOnly)

False

In [ ]:

### Note that the project file is read only if referenced more than once (e.g., script is run multiple times)
    # Most of the time project file will be read-only --- not allowing it to be saved after making changes 
    # More reliable to save it by making a copy of the initial project file

In [ ]:

### Use listMaps function on the ArcGISProject object to get a python list of the available maps in the project


# Set the map in the project file that you want to work on
my_proj_map = "Project Map"



# Store all map names in the project file
proj_maps= [m.name for m in aprx.listMaps()]
print(f"Available maps in the project file: ", proj_maps)


if my_proj_map in proj_maps:
    print(f"{my_proj_map} found\n")

    # Pick the map to load
    map_object = aprx.listMaps(my_proj_map)[0] ## Use arcpy.listMaps("MapName")[0] i.e., use an index value of 0 on the list to access the specific map object
    print("Successfully loaded the map")
else:
    print(f"{my_proj_map} not found in {map}")

Available maps in the project file:  ['Project Map', 'PCS_EqualEarth', 'NAD83', 'Mercator', 'Robinson', 'Albers', 'GCS_WGS84']
Project Map found

Successfully loaded the map

Add data to a to map¶

`map.addDataFromPath()`¶

Allows you to add data to a map in a project
addDataFromPath (data_path, {web_service_type}, {custom_parameters})
- can provide a feature class name, local path, or URL

In [123]:

# Store the fc in from the workspace
feature_classes =  arcpy.ListFeatureClasses()

print(gdb_path) # validate the path is the one you expect
print("\n")


### Load the fc onto the project map
try:

    for fc in feature_classes:
        fc_path = os.path.join(gdb_path,fc) # store that path to the fc
        map_object.addDataFromPath(fc_path)


    # Save the changes to the project file
    aprx.saveACopy(new_aprx_path)
    print("Successfully loaded the layers onto the project and saved the map")
    
except Exception as e:
    print("Error occured: ", e, type(e).__name__)
except arcpy.ExecuteError:
    print("Arcpy Error occured: ", arcpy.GetMessages(2))

# optional delete to avoid locks to the project
finally:
    del aprx

c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result\savageMN_BikePaths.gdb


Successfully loaded the layers onto the project and saved the map

Analysis of bike paths¶

In [ ]:

### Let's see if we can determine the total length of all bike paths 

fc_list = arcpy.ListFeatureClasses()

# Get the fc using a shortened name 
# helpful if we cannot remember the full fc name

for fc in fc_list:
    if "bike" in fc.lower():
        bikepaths=fc
        print(f"{bikepaths} Feature Class Found")
    
    elif "city" in fc.lower():
        ctu=fc
        print(f"{ctu} Feature Class Found")
    
    else:
        print("Feature Class not found")

Bikeways_in_Minnesota Feature Class Found
city_township_unorg Feature Class Found

In [110]:

### Check the fields for the bikepaths layer

cat.showFieldinfo(bikepaths)

*Summary*
 Fields in the Polyline Feature Class: Bikeways_in_Minnesota

Name                 Type         Length   Unique_Values
 OBJECTID             | OID        | 4     | 46735     
 Shape                | Geometry   | 0     | 46427     
 UNIQUE_ID            | String     | 36    | 46735     
 BKWYNAME             | String     | 150   | 2698      
 BKWYSYSTEM           | String     | 150   | 231       
 SHAREDNAME           | String     | 150   | 510       
 FACTYPE              | String     | 100   | 20        
 FED_SYS              | String     | 10    | 3         
 NATION_SYS           | String     | 10    | 3         
 STATE_SYS            | String     | 10    | 3         
 REGION_SYS           | String     | 10    | 3         
 COUNTY_SYS           | String     | 10    | 3         
 LOCAL_SYS            | String     | 10    | 3         
 TRIBAL_SYS           | String     | 10    | 3         
 PRIV_SYS             | String     | 10    | 3         
 LANDOWNER            | String     | 150   | 143       
 OWNERTYPE            | String     | 50    | 11        
 ORGNAME              | String     | 150   | 233       
 ORGTYPE              | String     | 50    | 13        
 BKWYSTATUS           | String     | 50    | 11        
 BKWYSURF             | String     | 50    | 9         
 YEAR_PRGRM           | Integer    | 4     | 67        
 YEAR_OPEN            | Integer    | 4     | 72        
 WIDTH_FT             | Double     | 8     | 35        
 SEASNL_ACC           | String     | 20    | 5         
 DIRECTION            | String     | 20    | 5         
 PVMNTMARKS           | String     | 10    | 4         
 LIGHTING             | String     | 10    | 4         
 SIGNING              | String     | 10    | 4         
 SEPARATION           | String     | 100   | 6         
 ROAD_BR              | String     | 100   | 4         
 RMBL_STRIP           | String     | 10    | 3         
 RMBL_TYPE            | String     | 100   | 2         
 RMBL_PLACE           | String     | 100   | 2         
 BKWY_URL             | String     | 254   | 97        
 DATASOURCE           | String     | 50    | 7         
 EDIT_ORG             | String     | 100   | 55        
 EDIT_DATE            | Date       | 8     | 51        
 COMMENTS             | String     | 254   | 680       
 SHAPE_Leng           | Double     | 8     | 46414     
 Shape_Length         | Double     | 8     | 46418     

Total Record Count: 46735, Total Geometry Count(by WKT): 46431
Warning 304 Potential Duplicate Features Detected in the Feature Class

In [ ]:

### Shape Length field is useful here

In [ ]:

### Search and sum the bike layer feature length for all features

total_pathLength= 0

with arcpy.da.SearchCursor(bikepaths,["Shape_Length"]) as cursor:
    for row in cursor:
        if row[0] is not None:
            total_pathLength += row[0]

print(f"Total bike path length across the state is: {round(total_pathLength,2)} meters or {round(total_pathLength* 0.00062137,2)} miles")

Total bike path length across the state is: 20925649.04 meters or 13002.57 miles

In [115]:

cat.showFieldinfo(ctu)

*Summary*
 Fields in the Polygon Feature Class: city_township_unorg

Name                 Type         Length   Unique_Values
 OBJECTID             | OID        | 4     | 2743      
 Shape                | Geometry   | 0     | 2743      
 GNIS_FEATU           | Integer    | 4     | 2693      
 FEATURE_NA           | String     | 254   | 2249      
 CTU_CLASS            | String     | 25    | 3         
 COUNTY_GNI           | Integer    | 4     | 87        
 COUNTY_COD           | String     | 2     | 87        
 COUNTY_NAM           | String     | 100   | 87        
 POPULATION           | Integer    | 4     | 1312      
 SHAPE_Leng           | Double     | 8     | 2743      
 Shape_Length         | Double     | 8     | 2743      
 Shape_Area           | Double     | 8     | 2743      

Total Record Count: 2743, Total Geometry Count(by WKT): 2743
Warning 0 Potential Duplicate Features Detected in the Feature Class

In [124]:

### Find the total bike paths lengths within a specific city

# We can use the field FEATURE_NA to access a specific city
# Then create layers to enable use to select bike paths by locations
# search and sum the bike layer feature length for all features



ctu_lyr= "ctu_layer"
bike_lyr = "bike_layer"
ctu_name=  "Savage"
# Select specific ctu
arcpy.management.MakeFeatureLayer(ctu,ctu_lyr,where_clause=f"FEATURE_NA = '{ctu_name}'")

# select bike paths that intersect the ctu
arcpy.management.MakeFeatureLayer(bikepaths,bike_lyr)


### arcpy.management.SelectLayerByLocation(in_layer, {overlap_type}, {select_features}, {search_distance}, {selection_type}, {invert_spatial_relationship})
arcpy.management.SelectLayerByLocation(bike_lyr,"INTERSECT",ctu_lyr)



ctu_pathLength= 0
with arcpy.da.SearchCursor(bike_lyr,["Shape_Length"]) as cursor:
    for row in cursor:
        if row[0] is not None:
            ctu_pathLength += row[0]

print(f"Total bike path length across the state is: {round(ctu_pathLength,2)} meters or {round(ctu_pathLength* 0.00062137,2)} miles")

Total bike path length across the state is: 183180.46 meters or 113.82 miles

In [ ]:

### There may be a better way to ensure we only count paths within CTU, such as using WITHIN as the overlap type

Summarize features within a polygon layer and compute statistics using Summarize Within¶

Overlays a polygon layer with another layer to summarize the number of points, length of lines, area of the polygons within each polygon
Computes field statistics for those features within the polygons
Can create groups by choosing a group field for the input features (e.g., summarize crimes within neighborhood polygons by a Crime_Type attribute) #### arcpy.analysis.SummarizeWithin()
arcpy.analysis.SummarizeWithin(in_polygons, in_sum_features, out_feature_class, {keep_all_polygons}, {sum_fields}, {sum_shape}, {shape_unit}, {group_field}, {add_min_maj}, {add_group_percent}, {out_group_table})
in_polygons: polygons to use for summarizing features
in_sum_features: point, line, or polygons that will be summarized for each polygon in the input polygons
out_feature_class: the output fc that will have the same geometry and attributes as the input polygon
- will include additional fields: number of points, length of lines, or area of polygons inside each input and statistics about the features
keep_all_polygons: whether to keep all or only the intersecting polygons with at least one input summary feature
- KEEP_ALL
- ONLY_INTERSECTING
sum_fields: list of attribute field names and statistical summary types [[summary_field, statistic_type]]
shape_unit: units to use when calculating shape summary attributes
- if input summary features is points, not necessary- only counts will be added
- if lines, specify linear unit (METERS, KILOMETERS, FEET, MILES, ACRES...)
- if polygons, specify areal unit ( SQUAREMETERS, SQUAREKILOMETERS, SQUAREFEET, SQUAREMILES)
group_field: field from input summary features to use for grouping, feature with the same value for the group field will be combined
add_min_maj: shows which group field value is most predominant (majority) and which is least predominant (minority) within each polygon\
add_group_percent: shows the percentage of each attribute value in each group

In [5]:

### set the wksp if not already done, list feature classes

wksp= arcpy.env.workspace = r"C:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS\02 Result\savageMN_BikePaths.gdb"
arcpy.ListFeatureClasses()

Out[5]:

['Bikeways_in_Minnesota', 'city_township_unorg']

In [6]:

import custom_arcpy_tools as cat
cat.showFieldinfo("Bikeways_in_Minnesota")

Initial working directory:  c:\Projects\my_git_pages_website\Py-and-Sky-Labs\content\GIS
*Summary*
 Fields in the Polyline Feature Class: Bikeways_in_Minnesota

Name                 Type         Length   Unique_Values
 OBJECTID             | OID        | 4     | 46735     
 Shape                | Geometry   | 0     | 46427     
 UNIQUE_ID            | String     | 36    | 46735     
 BKWYNAME             | String     | 150   | 2698      
 BKWYSYSTEM           | String     | 150   | 231       
 SHAREDNAME           | String     | 150   | 510       
 FACTYPE              | String     | 100   | 20        
 FED_SYS              | String     | 10    | 3         
 NATION_SYS           | String     | 10    | 3         
 STATE_SYS            | String     | 10    | 3         
 REGION_SYS           | String     | 10    | 3         
 COUNTY_SYS           | String     | 10    | 3         
 LOCAL_SYS            | String     | 10    | 3         
 TRIBAL_SYS           | String     | 10    | 3         
 PRIV_SYS             | String     | 10    | 3         
 LANDOWNER            | String     | 150   | 143       
 OWNERTYPE            | String     | 50    | 11        
 ORGNAME              | String     | 150   | 233       
 ORGTYPE              | String     | 50    | 13        
 BKWYSTATUS           | String     | 50    | 11        
 BKWYSURF             | String     | 50    | 9         
 YEAR_PRGRM           | Integer    | 4     | 67        
 YEAR_OPEN            | Integer    | 4     | 72        
 WIDTH_FT             | Double     | 8     | 35        
 SEASNL_ACC           | String     | 20    | 5         
 DIRECTION            | String     | 20    | 5         
 PVMNTMARKS           | String     | 10    | 4         
 LIGHTING             | String     | 10    | 4         
 SIGNING              | String     | 10    | 4         
 SEPARATION           | String     | 100   | 6         
 ROAD_BR              | String     | 100   | 4         
 RMBL_STRIP           | String     | 10    | 3         
 RMBL_TYPE            | String     | 100   | 2         
 RMBL_PLACE           | String     | 100   | 2         
 BKWY_URL             | String     | 254   | 97        
 DATASOURCE           | String     | 50    | 7         
 EDIT_ORG             | String     | 100   | 55        
 EDIT_DATE            | Date       | 8     | 51        
 COMMENTS             | String     | 254   | 680       
 SHAPE_Leng           | Double     | 8     | 46414     
 Shape_Length         | Double     | 8     | 46418     

Total Record Count: 46735, Total Geometry Count(by WKT): 46431
Warning 304 Potential Duplicate Features Detected in the Feature Class

In [7]:

cat.showFieldinfo("city_township_unorg")

*Summary*
 Fields in the Polygon Feature Class: city_township_unorg

Name                 Type         Length   Unique_Values
 OBJECTID             | OID        | 4     | 2743      
 Shape                | Geometry   | 0     | 2743      
 GNIS_FEATU           | Integer    | 4     | 2693      
 FEATURE_NA           | String     | 254   | 2249      
 CTU_CLASS            | String     | 25    | 3         
 COUNTY_GNI           | Integer    | 4     | 87        
 COUNTY_COD           | String     | 2     | 87        
 COUNTY_NAM           | String     | 100   | 87        
 POPULATION           | Integer    | 4     | 1312      
 SHAPE_Leng           | Double     | 8     | 2743      
 Shape_Length         | Double     | 8     | 2743      
 Shape_Area           | Double     | 8     | 2743      

Total Record Count: 2743, Total Geometry Count(by WKT): 2743
Warning 0 Potential Duplicate Features Detected in the Feature Class

In [ ]:

### Use the Summarize within tool to group cycling routes in Minnesota by city township polygons

inpoly = 'city_township_unorg'
infc= 'Bikeways_in_Minnesota'
outfc= 'SummarizedBikePaths_ctu'
keepAll= "KEEP_ALL"
sumFields= [["Shape_Length","SUM"]]
addShapeSum = 'ADD_SHAPE_SUM'
sumUnits= "MILES"
groupField= "FACTYPE"
addMinMaj = 'ADD_MIN_MAJ'
addPercents = 'ADD_PERCENT'
outTable = 'bikePaths_combined_groups'


try:
    arcpy.analysis.SummarizeWithin(inpoly, infc, outfc, keepAll, sumFields,addShapeSum,sumUnits, groupField, addMinMaj, addPercents,outTable)
    print("Successfully ran the summarize within tool")

except Exception as e:
    print("Error occured", e, type(e).__name__)
except arcpy.ExecuteError:
    print("Arcpy error", arcpy.GetMessages(2))

Successfully ran the summarize within tool

Savage, MN GIS

Other articles

GIS

Real-time Drone Folium Maps GIS

How to Use¶

Introduction to Folium Maps GIS

Make a folium map of drone data¶

Other articles

How to Use¶

Make a folium map of drone data¶

links

social