Prediction of energy generation from Solar Photovoltaic Power Plants using weather variables

Introduction

Recently there has been a great emphasis on reducing carbon footprint by moving away from fossil fuel to renewable energy sources for running our cities. Various local city governments across the world like in this case the City of Calgary in Canada is leading this change by becoming energy independent by installing solar power plants either on rooftops or within the site area of their city utilities for running its operation.

In view of the scenario here is a notebook that would predict the daily hence annual solar energy generation by a solar power station at a site using local weather information and site characteristics. The hypothesis is that various weather parameters such as temperature, wind speed, vapor pressure, solar radiation, day length, precipitation, snowfall along with altitude of a place would impact the daily generation of solar energy.

Accordingly, these variables are used to train a model on actual solar power generated by solar stations located in Calgary, which could then be used to predict solar generation for probable solar plants at other locations. Besides the total energy generation would also depend on the capacity of the solar station established. For example, a 100kwp solar plant will generate more energy than a 50kwp plant, hence for the final output, the capacity of the plant is to be taken into consideration.

Imports

%matplotlib inline
import matplotlib.pyplot as plt

import pandas as pd
from pandas import read_csv
from datetime import datetime
from IPython.display import Image, HTML

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Normalizer
from sklearn.preprocessing import MinMaxScaler  
from sklearn.compose import make_column_transformer 
from sklearn.metrics import r2_score

import arcgis
from arcgis.gis import GIS
from arcgis.learn import FullyConnectedNetwork, MLModel, prepare_tabulardata

Connecting to ArcGIS

gis = GIS(profile="your_online_profile")

Accessing & Visualizing datasets

The primary data used for this sample are as follows:

Out of the several solar photovoltaic power plants in the City of Calgary, 11 were selected for the study. The dataset contains two components:

1) Daily solar energy production for each power plant from September 2015 to December 2019.

2) Corresponding daily weather measurements for the given sites.

The datasets were obtained from multiple sources as mentioned here (Data resources) and preprocessed to obtain the main dataset used here. Two feature layers was subsequently created out of them one for training and the other for validating.

Training Set

It consists of data from 10 solar sites for training the model. The feature layer containing the data is accessed here from Arcgis portal and visualized as follows:

# Access Solar Dataset feature layer for Training, without the Southland Solar Plant which is hold out for validation
calgary_no_southland_solar = gis.content.search('calgary_no_southland_solar owner:api_data_owner', 'feature layer')[0]
calgary_no_southland_solar
calgary_no_southland_solar
Feature Layer Collection by api_data_owner
Last Modified: October 23, 2020
0 comments, 11650 views
# Access the layer from the feature layer
calgary_no_southland_solar_layer = calgary_no_southland_solar.layers[0]
# Plot location of the 10 Solar sites in Calgary to be used for training
m1 = gis.map('calgary', zoomlevel=10)
m1.add_layer(calgary_no_southland_solar_layer)
m1

The map above shows the 10 power plant locations that are used for collecting the training data.

# Visualize the dataframe
calgary_no_southland_solar_layer_sdf = calgary_no_southland_solar_layer.query().sdf
calgary_no_southland_solar_layer_sdf=calgary_no_southland_solar_layer_sdf[['FID','date','ID','solar_plan','altitude_m',
                                                                           'latitude','longitude','wind_speed','dayl__s_',
                                                                           'prcp__mm_d','srad__W_m_','swe__kg_m_', 'tmax__deg',
                                                                           'tmin__deg','vp__Pa_','kWh_filled','capacity_f',
                                                                           'SHAPE']]
calgary_no_southland_solar_layer_sdf.head()
FIDdateIDsolar_planaltitude_mlatitudelongitudewind_speeddayl__s_prcp__mm_dsrad__W_m_swe__kg_m_tmax__degtmin__degvp__Pa_kWh_filledcapacity_fSHAPE
012017-12-24355827Glenmore Water Treatment Plant109551.003078-114.1005717.2046727648.01108.80000312-10.5-21.01201.2423570.000177{"x": -12701617.407282012, "y": 6621838.159138...
122017-12-25355827Glenmore Water Treatment Plant109551.003078-114.1005713.38523527648.01115.19999712-18.0-29.5402.4777140.000354{"x": -12701617.407282012, "y": 6621838.159138...
232017-12-26355827Glenmore Water Treatment Plant109551.003078-114.1005715.07631627648.00118.40000212-20.0-32.0403.7130710.00053{"x": -12701617.407282012, "y": 6621838.159138...
342017-12-27355827Glenmore Water Treatment Plant109551.003078-114.1005715.61762327648.0096.012-18.0-26.5804.9484290.000707{"x": -12701617.407282012, "y": 6621838.159138...
452017-12-28355827Glenmore Water Treatment Plant109551.003078-114.1005712.56151227648.00118.40000212-17.0-28.5406.1837860.000883{"x": -12701617.407282012, "y": 6621838.159138...

In the above table, each row represents each day starting from September 2015 to December 2019, with the corresponding dates shown in the column named date, while the field solar_plan contains name of the solar sites.

The primary information consists of the daily generation of energy in kilowatt-hour(KWh) given here in the field name kWh_filled for each of the selected 10 solar photovoltaic power plants in the City of Calgary. The field capacity_f indicates the capacity factor which is obtained after normalizing the kWh_filled by the peak capacity of each solar photovoltaic sites, which will be used here as the dependent variable.

In addition it contains data about weather variables for each day for the related solar plant, all of which except wind speed, was obtained from MODIS, Daymet observations. These variables are as follows:

  • wind_speed : wind speed(m/sec)
  • dayl_s : Day length (sec/day)
  • prcp__mm_d : Precipitation (mm/day)
  • srad_W_m : Shortwave radiation (W/m^2)
  • swe_kg_m : Snow water equivalent (kg/m^2)
  • tmax__deg : Maximum air temperature (degrees C)
  • tmin__deg : Minimum air temperature (degrees C)
  • vp_Pa : Water vapor pressure (Pa)

Now to understand the distribution of the variables over the last few years and their respective relationship with the dependent variable of daily energy produced for that stations, data from one of the station is plotted in the following.

# plot and Visualize the variables from the training set for one solar station - Hillhurst Sunnyside Community Association 
hillhurst_solar = calgary_no_southland_solar_layer_sdf[calgary_no_southland_solar_layer_sdf['solar_plan']=='Hillhurst Sunnyside Community Association'].copy()
hillhurst_datetime = hillhurst_solar.set_index(hillhurst_solar['date'])
hillhurst_datetime = hillhurst_datetime.sort_index() 
for i in range(7,hillhurst_datetime.shape[1]-1):
        plt.figure(figsize=(20,3))
        plt.title(hillhurst_datetime.columns[i])
        plt.plot(hillhurst_datetime[hillhurst_datetime.columns[i]])
        plt.show()
<Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes><Figure size 2000x300 with 1 Axes>

In the above plots it can be seen that each of the variables has high seasonality and it seems that there is some relationship between the dependent variable of kWh_filled and the rest. Hence this is followed by creating a correlation plot to check the correlation between the variables.

# checking the correlation matrix between the predictors and the dependent variable of capacity_factor
corr_test = calgary_no_southland_solar_layer_sdf.drop(['FID','date','ID','latitude','longitude','solar_plan','kWh_filled','SHAPE'], axis=1)
corr = corr_test.corr()
corr.style.background_gradient(cmap='Greens').format(precision=2)
 altitude_mwind_speeddayl__s_prcp__mm_dsrad__W_m_swe__kg_m_tmax__degtmin__degvp__Pa_capacity_f
altitude_m1.00-0.010.040.010.030.020.020.020.020.03
wind_speed-0.011.00-0.41-0.17-0.260.02-0.03-0.06-0.13-0.24
dayl__s_0.04-0.411.000.200.78-0.180.720.730.600.77
prcp__mm_d0.01-0.170.201.00-0.18-0.07-0.030.100.20-0.04
srad__W_m_0.03-0.260.78-0.181.000.040.690.500.280.82
swe__kg_m_0.020.02-0.18-0.070.041.00-0.45-0.48-0.46-0.19
tmax__deg0.02-0.030.72-0.030.69-0.451.000.930.750.75
tmin__deg0.02-0.060.730.100.50-0.480.931.000.850.65
vp__Pa_0.02-0.130.600.200.28-0.460.750.851.000.45
capacity_f0.03-0.240.77-0.040.82-0.190.750.650.451.00

The plot shows that the variable of shortwave radiation per meter square (srad_W_m) received at the site has the maximum correlation with the dependent variable of total solar energy produced expressed in terms of capacity factor(capacity_f), which is self-explanatory. This is followed by the variable of day length(dayl_s) which means that longer the day more the produced energy. These two are closely followed by max(tmax__deg) and min(tmin__deg) daily temperatures, and lastly the other variables.

Validation Set

This set consists of daily solar generation data dated from September, 2015 to December, 2019 of one solar site known as Southland Leisure Centre for the purpose of validating the trained model:-

# Access the Southland Solar Plant Dataset feature layer for validation
southland_solar = gis.content.search('southland_solar owner:api_data_owner', 'feature layer')[0]
southland_solar
southland_solar
Feature Layer Collection by api_data_owner
Last Modified: April 27, 2020
0 comments, 428 views
# Access the layer from the feature layer
southland_solar_layer = southland_solar.layers[0]
#  Plot location of the Southalnd Solar site in Calgary to be used for validation
m1 = gis.map('calgary', zoomlevel=10)
m1.add_layer(southland_solar_layer)
m1
# visualize the southland dataframe here
southland_solar_layer_sdf = southland_solar_layer.query().sdf
southland_solar_layer_sdf.head(2)
FIDField1IDsolar_planaltitude_mlatitudelongitudewind_speeddayl__s_prcp__mm_d...tmin__degvp__Pa_kWh_filledcapacity_fGlobalIDCreationDateCreatorEditDateEditorSHAPE
012019-10-03164440Southland Leisure Centre1100.050.962485-114.1084725.33223940089.6015620.0...-3.0480.0309.6440.084326e9b0f671-d6ba-4560-b912-d635a0a129f82020-04-27 11:58:02.992arcgis_python2020-04-27 11:58:02.992arcgis_python{"x": -12702497.020502415, "y": 6614660.374377...
122019-10-04164440Southland Leisure Centre1100.050.962485-114.1084726.30482940089.6015620.0...-1.0560.0679.7850.1851277bde5210-a8c2-4731-9c23-e5f77c1ebc562020-04-27 11:58:02.992arcgis_python2020-04-27 11:58:02.992arcgis_python{"x": -12702497.020502415, "y": 6614660.374377...

2 rows × 23 columns

# check the total number of samples
southland_solar_layer_sdf.shape
(1590, 23)

Model Building

Once the training and the validation dataset is processed and analyzed, it is ready to be used for modeling.

In this sample two types of methodology are used for modeling:

1) FullyConnectedNetwork - First a deep learning framework called FullyConnectedNetwork available in the arcgis.learn module in ArcGIS API for Python is used.

2) MLModel - In the second option, a regression model from scikit-learn is implemented via the MLModel framework in arcgis.learn. This framework can deploy any regression or classification model from the library just by passing the name of the algorithm and its relevant parameters as keyword arguments.

Finally, performance between the two methods will be compared in terms of model training and validation accuracy.

Further details on FullyConnectedNetwork & MLModel are available here in the Deep Learning with ArcGIS section.

1 — FullyConnectedNetwork

This is an Artificial Neural Network model from the arcgis.learn module which is used here for modeling.

Data Preprocessing

First a list is made consisting of the feature data that will be used for predicting daily solar energy generation. By default, it will receive continuous variables, while in case of categorical variables the True value should be passed inside a tuple along with the variable. Here all the variables are continuous.

# Here a list is created naming all fields containing the predictors from the input feature layer
X = ['altitude_m', 'wind_speed', 'dayl__s_', 'prcp__mm_d','srad__W_m_','swe__kg_m_','tmax__deg','tmin__deg','vp__Pa_']
# importing the libraries from arcgis.learn for data preprocessing
from arcgis.learn import prepare_tabulardata

Once the explanatory variables are identified the main preprocessing of the data is carried out by the prepare_tabulardata method from the arcgis.learn module in the ArcGIS API for Python. The function takes a feature layer or a spatial dataframe containing the dataset as input and returns a TabularDataObject that can be fed into the model.

The input parameters required for the tool are:

  • input_features : feature layer or spatial dataframe having the primary dataset
  • variable_predict : field name containing the y-variable from the input feature layer/dataframe
  • explanatory_variables : list of the field names as 2-sized tuples containing the explanatory variables as mentioned above
# precrocessing data using prepare data method - it handles imputing missing values, normalization and train-test split
data = prepare_tabulardata(calgary_no_southland_solar_layer,
                           'capacity_f',
                           explanatory_variables=X)
# visualizing the prepared data 
data.show_batch()
altitude_mcapacity_fdayl__s_prcp__mm_dsrad__W_m_swe__kg_m_tmax__degtmin__degvp__Pa_wind_speed
164010550.10661237324.8007810281.600006018.53.58006.391703
178810950.07081355641.6015627144.007.52.57205.120847
182510950.22494858752.01387.200012020.07.58803.322512
414010700.01627940780.8007810300.79998836-11.0-19.51203.128044
789710960.23741150457.6015620441.600006027.04.06804.591178

Model Initialization

Once the data has been prepared by the prepare_tabulardata method it is ready to be passed to the ANN for training. First the ANN known as FullyConnectedNetwork is imported from arcgis.learn and initialized as follows:

# importing the model from arcgis.learn
from arcgis.learn import FullyConnectedNetwork
# Initialize the model with the data where the weights are randomly allocated 
fcn = FullyConnectedNetwork(data, layers=[200,130])
# searching for an optimal learning rate using the lr_find for passing it to the final model fitting 
lr = fcn.lr_find()
lr
<Figure size 640x480 with 1 Axes>
0.0005754399373371565

Here the suggested learning rate by the lr_find method is around 0.0012. The automatic lr_finder will take a conservative estimate of the learning rate, but some experts can interpret the graph more appropriately and find a better learning rate to be used for final training of the model.

Model Training

Finally the model is now ready for training, and the model.fit method is used which is given the number of epochs for training and the estimated learning rate selected based on the lr_find returned in the previous step:

# the model is trained for 100 epochs 
fcn.fit(100, lr=lr)
epochtrain_lossvalid_losstime
00.0039930.00282600:00
10.0026350.00245300:00
20.0024270.00226500:00
30.0022330.00216800:00
40.0021870.00214700:00
50.0022350.00203500:00
60.0020450.00198700:00
70.0021120.00199000:00
80.0020570.00192600:00
90.0019780.00188100:00
100.0020290.00188000:00
110.0020240.00198700:00
120.0020150.00185000:00
130.0020480.00191400:00
140.0019910.00190400:00
150.0019820.00188700:00
160.0020810.00207100:00
170.0020400.00181700:00
180.0020700.00190400:00
190.0021460.00181100:00
200.0019040.00184100:00
210.0019590.00196800:00
220.0020090.00197000:00
230.0020220.00244200:00
240.0019460.00183400:00
250.0019120.00178400:00
260.0018920.00191300:00
270.0019040.00174700:00
280.0017980.00190800:00
290.0018910.00175000:00
300.0018890.00180700:00
310.0017880.00169600:00
320.0017620.00175200:00
330.0017810.00176400:00
340.0018350.00188000:00
350.0018210.00174600:00
360.0017950.00169700:00
370.0017640.00169100:00
380.0017360.00166800:00
390.0017410.00177900:00
400.0017390.00172500:00
410.0016730.00160300:00
420.0017820.00169900:00
430.0016850.00160400:00
440.0016850.00161400:00
450.0016840.00167400:00
460.0016910.00167600:00
470.0016380.00161900:00
480.0016060.00160700:00
490.0016500.00161400:00
500.0015670.00162800:00
510.0015890.00160800:00
520.0016300.00153500:00
530.0016180.00155300:00
540.0015700.00165000:00
550.0015760.00160200:00
560.0015170.00169200:00
570.0015900.00158300:00
580.0015360.00154900:00
590.0015520.00153900:00
600.0014900.00150700:00
610.0015390.00152900:00
620.0015350.00151600:00
630.0015190.00152300:00
640.0014900.00152300:00
650.0014230.00151100:00
660.0015330.00170700:00
670.0014600.00152100:00
680.0014510.00151900:00
690.0014920.00148700:00
700.0014870.00149600:00
710.0014550.00146700:00
720.0014560.00146900:00
730.0014270.00149100:00
740.0014510.00144900:00
750.0013850.00145800:00
760.0014340.00146600:00
770.0014840.00148700:00
780.0014060.00148400:00
790.0014420.00148400:00
800.0013550.00147500:00
810.0014110.00149900:00
820.0013670.00146600:00
830.0014460.00146600:00
840.0013960.00146700:00
850.0013990.00146200:00
860.0013300.00145100:00
870.0014010.00147300:00
880.0013930.00144400:00
890.0013130.00144300:00
900.0013500.00147200:00
910.0013390.00146600:00
920.0013390.00146800:00
930.0014040.00143700:00
940.0013130.00149700:00
950.0013410.00147000:00
960.0013460.00149400:00
970.0013170.00145500:00
980.0013340.00145900:00
990.0013730.00149200:00
100.00% [5/5 00:00<00:00]

The train vs valid losses is plotted to check if the model is overfitting. It shows that the model has trained well and though the losses are still gradually decreasing but not significantly.

# the train vs valid losses is plotted to check quality of the trained model
fcn.plot_losses()
<Figure size 640x480 with 1 Axes>

Finally, the training results are printed to assess the prediction on the test set by the trained model.

# the predicted values by the trained model is printed for the test set
fcn.show_results()
100.00% [5/5 00:00<00:00]
altitude_mcapacity_fdayl__s_prcp__mm_dsrad__W_m_swe__kg_m_tmax__degtmin__degvp__Pa_wind_speedprediction_results
315010950.25661658406.3984380480.0022.06.05203.5702360.205817
15810950.12095535942.3984380240.0013.0-3.04807.9644950.124888
592710700.28804458752.02470.399994023.06.55603.3395480.243402
742810940.0834358752.023201.600006020.512.513606.2241270.111588
890010510.00717141126.3984381141.5999980-1.5-3.04807.4125530.001019

In the above table, the predicted values by the model on the test set in the last column named prediction_results and the actual values in the column named capacity_f of the target variable are highly similar.

Accordingly, the model metrics of the trained model is now estimated using the model.score function. It returns the r-square of the model fit as follows:

# the model.score method from the tabular learner returns r-square
r_Square_fcn_test = fcn.score() 
print('r_Square_fcn_test: ', round(r_Square_fcn_test,5))
100.00% [912/912 00:07<00:00]
r_Square_fcn_test:  0.84108

The high r-square value indicates that the model has been trained well

Solar Energy Generation Forecast & Validation

The trained model(FullyConnectedNetwork) will now be used to predict the daily lifetime solar energy generation for the solar plant installed at the Southland Leisure Centre since it was installed during 2015. The aim is to validate the trained model and measure its performance of solar output estimation using only weather variables from the Southland Leisure Center.

Accordingly the model.predict method from arcgis.learn is used with the daily weather variables as input for the mentioned site ranging from September, 2015 to December, 2019 to predict daily solar energy output in KWh for the same time period. The predictors are automatically chosen from the input feature layer of southland_layer by the trained model without mentioning them explicitly, since their names are exactly same as used during training the model.

# predicting using the predict function 
southland_solar_layer_predicted = fcn.predict(southland_solar_layer, output_layer_name='prediction_layer')
100.00% [1590/1590 00:12<00:00]
# print the predicted layer
southland_solar_layer_predicted
prediction_layer
Feature Layer Collection by api_data_owner
Last Modified: July 29, 2024
0 comments, 0 views
# Access & visualize the dataframe from the predicted layer 
test_pred_layer = southland_solar_layer_predicted.layers[0]
test_pred_layer_sdf = test_pred_layer.query().sdf
test_pred_layer_sdf.head()
FIDFID_1Field1IDsolar_planaltitude_mlatitudelongitudewind_speeddayl__s_...CreatorEditDateEditorzone3_idzone4_idzone5_idzone6_idzone7_idpredictionSHAPE
011201.02018-09-10164440Southland Leisure Centre1100.050.962485-114.1084723.7146245619.199219...arcgis_python2020-04-27arcgis_python8312ccfffffffff8412ccdffffffff8512ccc3fffffff8612ccd57ffffff8712ccd52ffffff0.129638{"x": -12702497.020502415, "y": 6614660.374377...
121202.02018-09-11164440Southland Leisure Centre1100.050.962485-114.1084723.66326245619.199219...arcgis_python2020-04-27arcgis_python8312ccfffffffff8412ccdffffffff8512ccc3fffffff8612ccd57ffffff8712ccd52ffffff0.149016{"x": -12702497.020502415, "y": 6614660.374377...
231203.02018-09-12164440Southland Leisure Centre1100.050.962485-114.1084723.84784745273.601562...arcgis_python2020-04-27arcgis_python8312ccfffffffff8412ccdffffffff8512ccc3fffffff8612ccd57ffffff8712ccd52ffffff0.0457{"x": -12702497.020502415, "y": 6614660.374377...
341204.02018-09-13164440Southland Leisure Centre1100.050.962485-114.1084723.95823644928.0...arcgis_python2020-04-27arcgis_python8312ccfffffffff8412ccdffffffff8512ccc3fffffff8612ccd57ffffff8712ccd52ffffff0.04043{"x": -12702497.020502415, "y": 6614660.374377...
451205.02018-09-14164440Southland Leisure Centre1100.050.962485-114.1084724.27544944582.398438...arcgis_python2020-04-27arcgis_python8312ccfffffffff8412ccdffffffff8512ccc3fffffff8612ccd57ffffff8712ccd52ffffff0.046646{"x": -12702497.020502415, "y": 6614660.374377...

5 rows × 30 columns

test_pred_layer_sdf.shape
(1590, 30)

The table above returns the predicted values for the Southland photovoltaic power plant stored in the field called prediction_results which has the model estimated daily capacity factor of energy generation, whereas the actual capacity factor is in the field named capacity_f.

The capacity factor is a normalized value which is now rescaled back to the original unit of KWh in the following, using the peak capacity of the Southland photovoltaic power plant which is 153KWp.

test_pred_layer_sdf.columns
Index(['FID', 'FID_1', 'Field1', 'ID', 'solar_plan', 'altitude_m', 'latitude',
       'longitude', 'wind_speed', 'dayl__s_', 'prcp__mm_d', 'srad__W_m_',
       'swe__kg_m_', 'tmax__deg', 'tmin__deg', 'vp__Pa_', 'kWh_filled',
       'capacity_f', 'GlobalID', 'CreationDa', 'Creator', 'EditDate', 'Editor',
       'zone3_id', 'zone4_id', 'zone5_id', 'zone6_id', 'zone7_id',
       'prediction', 'SHAPE'],
      dtype='object')
optional_columns = ['prediction_results','prediction']

pred_col = None 
for opt_col in optional_columns:
    if opt_col in test_pred_layer_sdf.columns:
        pred_col = opt_col
        break
# inverse scaling from capcacity factor to actual generation in KWh  - peak capcity of Southland Leisure Centre is 153KWp
test_pred_datetime = test_pred_layer_sdf[['Field1','capacity_f',pred_col]].copy()
test_pred_datetime = test_pred_datetime.rename(columns={'Field1':'date'})
test_pred_datetime['date'] = pd.to_datetime(test_pred_datetime['date']) 
test_pred_datetime = test_pred_datetime.set_index(test_pred_datetime['date'])
test_pred_datetime['Actual_generation(KWh)'] = test_pred_datetime['capacity_f']*24*153
test_pred_datetime['predicted_generation(KWh)'] = test_pred_datetime[pred_col]*24*153
test_pred_datetime = test_pred_datetime.drop(['date','capacity_f',pred_col], axis=1).sort_index() 
test_pred_datetime
Actual_generation(KWh)predicted_generation(KWh)
date
2015-09-01286.013645.847283
2015-09-02681.646573.81391
2015-09-03647.906550.952354
2015-09-04102.448188.131957
2015-09-0593.432105.872621
.........
2019-12-271.34940.980099
2019-12-281.96522.588846
2019-12-291.61674.11202
2019-12-307.44110.594964
2019-12-318.32357.998556

1590 rows × 2 columns

The table above shows the actual versus the model predicted daily solar energy generated for the Southland plant for the duration of late 2015 to the end of 2019. These values are now used to estimate the various model metrics to understand the prediction power of the model.

# estimate model metrics of r-square, rmse and mse for the actual and predicted values for daily energy generation
from sklearn.metrics import r2_score
r2_test = r2_score(test_pred_datetime['Actual_generation(KWh)'],test_pred_datetime['predicted_generation(KWh)'])
print('R-Square: ', round(r2_test, 2))
R-Square:  0.86

The comparison returns a considerably high r-square of 0.86 showing high similarity between actual and predicted values.

# Comparison between the actual sum of the total energy generated to the total predicted values 
actual = (test_pred_datetime['Actual_generation(KWh)'].sum()/4/1000).round(2)  
predicted = (test_pred_datetime['predicted_generation(KWh)'].sum()/4/1000).round(2) 
print('Actual annual Solar Energy Generated by Southland Solar Station: {} MWh'.format(actual))
print('Predicted annual Solar Energy Generated by Southland Solar Stations: {} MWh'.format(predicted))
Actual annual Solar Energy Generated by Southland Solar Station: 170.03 MWh
Predicted annual Solar Energy Generated by Southland Solar Stations: 170.44 MWh

Result Visualization

Finally, the actual and predicted values are plotted to visualize their distribution across the entire lifetime of the power plant.

plt.figure(figsize=(30,6))
plt.plot(test_pred_datetime['Actual_generation(KWh)'],  linewidth=1, label= 'Actual') 
plt.plot(test_pred_datetime['predicted_generation(KWh)'], linewidth=1, label= 'Predicted')  
plt.ylabel('Solar Energy in KWh', fontsize=14)
plt.legend(fontsize=14,loc='upper right')
plt.title('Actual Vs Predicted Solar Energy Generated by Southland Solar-FulyConnectedNetwork Model', fontsize=14)
plt.grid()
plt.show() 
<Figure size 3000x600 with 1 Axes>

Summarizing the values, it is seen that the actual average annual energy generated by the solar plant is very close to the predicted annual average generated energy which reveals high precision.

In the plot above the blue line indicates the actual generation and the orange line shows the predicted values, both of which overlaps each other to a high degree, showing a high predictive capacity of the model.

MLModel

In the second methodology a machine learning model is applied to model the same data using the MLModel framework from arcgis.learn. This framework could be used to import and apply any machine learning model from the scikit-learn library on the data returned by the prepare_tabulardata function from arcgis.learn.

# importing the libraries from arcgis.learn for data preprocessing for the Machine Learning Model
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Normalizer
from sklearn.preprocessing import MinMaxScaler  
from sklearn.compose import make_column_transformer 

Data Preprocessing

Like the data preparation process for the neural network, first a list is made consisting of the feature data that will be used for predicting daily solar energy generation. By default, it will receive continuous variables, whereas for categorical variables the True value should be passed inside a tuple along with the variables. These variables are then transformed by the RobustScaler function from scikit-learn by passing it along with the variable list into the column transformer function as follows:

# scaling the feature data using MinMaxScaler(), the default is Normalizer from scikit learn
X = ['altitude_m', 'wind_speed', 'dayl__s_', 'prcp__mm_d','srad__W_m_','swe__kg_m_','tmax__deg','tmin__deg','vp__Pa_']
numerical_transformer = make_pipeline(MinMaxScaler())
preprocessors = make_column_transformer((numerical_transformer, X))

Once the explanatory variables list is defined and the precrocessors are computed these are now used as input for the prepare_tabulardata method in arcgis.learn. The method takes a feature layer or a spatial dataframe containing the dataset and returns a TabularDataObject that can be fed into the model.

The input parameters required for the tool are similar to the ones mentioned previously:

# importing the library from arcgis.learn for prepare data
from arcgis.learn import prepare_tabulardata
# precrocessing data using prepare data method for MLModel
data = prepare_tabulardata(calgary_no_southland_solar_layer,
                           'capacity_f',
                           explanatory_variables=X,                           
                           preprocessors=preprocessors)

Model Initialization

Once the data has been prepared by the prepare_tabulardata method it is ready to be passed to the selected machine learning model for training. Here the GradientBoostingRegressor model from scikit-learn is used which is passed into the MLModel function, along with its parameters as follows:

# importing the MLModel framework from arcgis.learn and the model from scikit learn 
from arcgis.learn import MLModel

# defining the model along with the parameters 
model = MLModel(data, 'sklearn.ensemble.GradientBoostingRegressor',loss ='absolute_error', learning_rate=0.02, n_estimators=117, random_state=43)

Model Training

Finally, the model is now ready for training, and the model.fit method is used for fitting the machine learning model with its defined parameters mentioned in the previous step.

model.fit()

The training results are printed to compute some model metrics and assess the quality of the trained model.

model.show_results()
altitude_mcapacity_fdayl__s_prcp__mm_dsrad__W_m_swe__kg_m_tmax__degtmin__degvp__Pa_wind_speedcapacity_f_results
148910550.01955529376.0096.00-4.5-12.02405.8191280.015684
350211120.25301553568.00473.600006024.58.06805.0978130.227703
430410700.24806150112.00422.399994029.07.58003.7336510.213505
549110900.01859734905.6015620265.600006283.0-14.52008.4353820.047954
767910960.11201544582.3984380288.0022.510.512804.8868890.152136

In the above table the last column named capacity_f_results returns the predicted values by the model on the test set which is highly similar to the actual values in the column named capacity_f for the target variable.

Subsequently, the model metrics of the trained model is now estimated using the model.score() function which currently returns the r-square of the model fit as follows:

# r-square is estimated using the inbuilt model.score() from the tabular learner
print('r_square_test_rf: ', round(model.score(), 5))
r_square_test_rf:  0.76784

The high R-squared value indicates that the model has been trained well.

feature_imp_RF = model.feature_importances_
<Figure size 800x510 with 1 Axes>

Solar Energy Generation Forecast & Validation

The trained GradientBoostingRegressor model implemented via the MLModel will now be used to predict the daily lifetime solar energy generation for the solar plant installed at the Southland Leisure Centre similarly since it was installed during 2015. The aim is to compare and validate its performance as obtained by the FullyConnectedNetwork model.

To recapitulate the model.predict method from arcgis.learn is used with the daily weather variables as input for the mentioned site ranging from September, 2015 to December, 2019 to predict daily solar energy output in KWh for the same time period. The predictors are automatically chosen from the input feature layer of southland_layer by the trained model without mentioning them explicitly, since their names are exactly same as are used for training the model.

southland_solar_layer_predicted_rf = model.predict(southland_solar_layer, output_layer_name='prediction_layer_rf')
# print the predicted layer
southland_solar_layer_predicted_rf
prediction_layer_rf
Feature Layer Collection by api_data_owner
Last Modified: July 24, 2024
0 comments, 0 views
# Access & visualize the dataframe from the predicted layer 
valid_pred_layer = southland_solar_layer_predicted_rf.layers[0]
valid_pred_layer_sdf = valid_pred_layer.query().sdf
valid_pred_layer_sdf.head()
FIDFID_1Field1IDsolar_planaltitude_mlatitudelongitudewind_speeddayl__s_...vp__Pa_kWh_filledcapacity_fGlobalIDCreationDaCreatorEditDateEditorpredictionSHAPE
01401.02016-07-05164440Southland Leisure Centre1100.050.962485-114.1084723.78757158060.800781...920.0593.2750.1615673cbc6ed1-6504-4f41-8e4b-b95c628160702020-04-27arcgis_python2020-04-27arcgis_python0.15771{"x": -12702497.020502415, "y": 6614660.374377...
12402.02016-07-06164440Southland Leisure Centre1100.050.962485-114.1084723.3023158060.800781...920.0575.3970.156699d19858dc-1caa-4893-9e98-1540753eec4a2020-04-27arcgis_python2020-04-27arcgis_python0.206897{"x": -12702497.020502415, "y": 6614660.374377...
23403.02016-07-07164440Southland Leisure Centre1100.050.962485-114.1084723.92360958060.800781...880.0886.4230.2414010ede54fa-3541-45be-9b64-5cdf259f69aa2020-04-27arcgis_python2020-04-27arcgis_python0.222833{"x": -12702497.020502415, "y": 6614660.374377...
34404.02016-07-08164440Southland Leisure Centre1100.050.962485-114.1084724.3753157715.199219...1000.0976.1360.265832c7259240-55a6-40fe-8700-844fafc12b8f2020-04-27arcgis_python2020-04-27arcgis_python0.221457{"x": -12702497.020502415, "y": 6614660.374377...
45405.02016-07-09164440Southland Leisure Centre1100.050.962485-114.1084722.81672557715.199219...1000.0490.250.133516e59869a-e4e3-473a-b98f-a57ec6a5b4802020-04-27arcgis_python2020-04-27arcgis_python0.205337{"x": -12702497.020502415, "y": 6614660.374377...

5 rows × 25 columns

The table above returns the MLModel predicted values for the Southland plant stored in the field prediction whereas the actual capacity factor is in the field named capacity_f.

The capacity factor is a normalized value which is now rescaled back to the original unit of KWh in the following, using the peak capacity of the Southland photovoltaic power plant which is 153KWp.

valid_pred_layer_sdf.columns
Index(['FID', 'FID_1', 'Field1', 'ID', 'solar_plan', 'altitude_m', 'latitude',
       'longitude', 'wind_speed', 'dayl__s_', 'prcp__mm_d', 'srad__W_m_',
       'swe__kg_m_', 'tmax__deg', 'tmin__deg', 'vp__Pa_', 'kWh_filled',
       'capacity_f', 'GlobalID', 'CreationDa', 'Creator', 'EditDate', 'Editor',
       'prediction', 'SHAPE'],
      dtype='object')
# inverse scaling from capcacity factor to actual generation in KWh  - peak capcity of Southland Leisure Centre is 153KWp
valid_pred_datetime = valid_pred_layer_sdf[['Field1','capacity_f',pred_col]].copy()
valid_pred_datetime = valid_pred_datetime.rename(columns={'Field1':'date'})
valid_pred_datetime['date'] = pd.to_datetime(valid_pred_datetime['date']) 
valid_pred_datetime = valid_pred_datetime.set_index(valid_pred_datetime['date'])
valid_pred_datetime['Actual_generation(KWh)'] = valid_pred_datetime['capacity_f']*24*153
valid_pred_datetime['predicted_generation(KWh)'] = valid_pred_datetime[pred_col]*24*153
valid_pred_datetime = valid_pred_datetime.drop(['date','capacity_f',pred_col], axis=1)
valid_pred_datetime = valid_pred_datetime.sort_index() 
valid_pred_datetime.head()
Actual_generation(KWh)predicted_generation(KWh)
date
2015-09-01286.013736.742935
2015-09-02681.646673.932795
2015-09-03647.906516.336426
2015-09-04102.448200.73517
2015-09-0593.432179.647032

The table above shows the actual versus the MLModel predicted daily solar energy generated for the Southland plant for the duration of late 2015 to the end of 2019. These values are now used to estimate the various model metrics to understand the prediction power of the MLModel.

# estimate model metrics of r-square, rmse and mse for the actual and predicted values for daily energy generation
from sklearn.metrics import r2_score
r2_test = r2_score(valid_pred_datetime['Actual_generation(KWh)'],valid_pred_datetime['predicted_generation(KWh)'])
print('R-Square: ', round(r2_test, 2))
R-Square:  0.81

The comparison returns a considerably high R-squared showing high similarity between actual and predicted values.

# Comparison between the actual sum of the total energy generated to the total predicted values by the MLModel 
actual = (valid_pred_datetime['Actual_generation(KWh)'].sum()/4/1000).round(2)  
predicted = (valid_pred_datetime['predicted_generation(KWh)'].sum()/4/1000).round(2) 
print('Actual annual Solar Energy Generated by Southland Solar Station: {} MWh'.format(actual))
print('Predicted annual Solar Energy Generated by Southland Solar Stations: {} MWh'.format(predicted))
Actual annual Solar Energy Generated by Southland Solar Station: 170.03 MWh
Predicted annual Solar Energy Generated by Southland Solar Stations: 170.04 MWh

Summarizing the values, it is seen that the actual average annual energy generated by the solar plant is very close to the predicted annual average generated energy which reveals high precision.

Result Visualization

Finally, the actual and predicted values are plotted to visualize their distribution across the entire lifetime of the power plant.

plt.figure(figsize=(30,6))
plt.plot(valid_pred_datetime['Actual_generation(KWh)'],  linewidth=1, label= 'Actual') 
plt.plot(valid_pred_datetime['predicted_generation(KWh)'], linewidth=1, label= 'Predicted')  
plt.ylabel('Solar Energy in KWh', fontsize=14)
plt.legend(fontsize=14,loc='upper right')
plt.title('Actual Vs Predicted Solar Energy Generated by Southland Solar-FulyConnectedNetwork Model', fontsize=14)
plt.grid()
plt.show() 
<Figure size 3000x600 with 1 Axes>

Conclusion

The goal of the project is to create a model that could predict the daily solar energy efficiency hence actual output of a photovoltaic solar plant at a location using daily weather variables of the site as input, and thereby demonstrate the application of the newly implemented artificial neural network of FullyConnectedNetwork and machine learning models called MLModel available in the arcgis.learn module in ArcGIS API for Python.

Accordingly, data from 10 solar energy installation sites in the City of Calgary in Canada are used to train two different models — first the FullyConnectedNetwork model and second the MLModel framework from the arcgis.learn module. These were eventually used to predict the daily solar output of a different solar plant in Calgary which is held out from the training set. The steps for implementing these models are discussed and elaborated in the notebook including data preprocessing, model training and final inferencing.

Comparison of the result shows that both the models successfully predicted the solar energy output of the test solar plant with predicted values of 171.76 MWh and 171.51 MWh by the FullyConnectedNetwork and the MLModel algorithm respectively, compared to the actual value of average annual solar generation of 170.74 MWh for the station.

Finally going further, it would be interesting to apply this model on other solar generation plants located across different geographies and record its performances to understand the generalizability of the model.

Summary of methods used

MethodDescriptionExamples
prepare_tabulardataprepare data including imputation, normalization and train-test splitprepare data ready for fitting a MLModel or FullyConnectedNetwork model
FullyConnectedNetwork()set a fully connected neural network to a datainitialize a FullyConnectedNetwork model with prepared data
model.lr_find()find an optimal learning ratefinalize a good learning rate for training the FullyConnectedNetwork model
MLModel()select the ML algorithm to be used for fittingany regression or classification model from scikit learn can be used
model.fit()train a model with epochs & learning rate as inputtraining the FullyConnectedNetwork model with sutiable input
model.score()find the model metric of R-squared of the trained modelreturns R-squared value after training the MLModel and FullyConnectedNetwork model
model.predict()predict on a test setpredict values using the trained models on test input

Data resources

DatasetSourceLink
Calgary solar energyCalgary daily solar energy generationhttps://data.calgary.ca/Environment/Solar-Energy-Production/ytdn-2qsp
Calgary Photovoltaic SitesLocation of Calgary Solar sites in Lat & Lonhttps://data.calgary.ca/dataset/City-of-Calgary-Solar-Photovoltaic-Sites/vrdj-ycb5
Calgary Daily weather dataMODIS - Daily Surface Weather Data on a 1-km Grid for North America, Version 3https://daac.ornl.gov/DAYMET/guides/Daymet_V3_CFMosaics.html

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.