MCPcopy
hub / github.com/eldraco/domain_analyzer / fetch_file

Function fetch_file

crawler.py:1067–1105  ·  view source on GitHub ↗

This function fetchs a single file from a given URL

(url_to_fetch,directory)

Source from the content-addressed store, hash-verified

1065# FETCH FILE
1066############
1067def fetch_file(url_to_fetch,directory):
1068
1069 """
1070 This function fetchs a single file from a given URL
1071 """
1072
1073 global error_codes
1074
1075
1076 try:
1077 socket.setdefaulttimeout(50)
1078 web_file = urllib.request.urlopen(url_to_fetch.replace(' ','%20'))
1079
1080 try:
1081 local_file = directory+url_to_fetch.split('/')[-1]
1082 local_file = open(directory+url_to_fetch.split('/')[-1], 'w')
1083 except OSError as error:
1084 if 'File exists' in error:
1085 pass
1086 else:
1087 return -1
1088 local_file.write(web_file.read())
1089 local_file.close()
1090 return 1
1091 except urllib.error.HTTPError as error_code:
1092 try:
1093 print('\t\t\t\t> ({0})'.format(error_codes[str(error_code.getcode())]))
1094 except:
1095 print('\t\t\t\t> Error in requesting file')
1096 return -10
1097 except urllib.error.URLError as error_code:
1098 if debug:
1099 print('\t\t\t\t> ({0})'.format(error_code.reason))
1100 else:
1101 print('\t\t\t\t> ({0})'.format(error_code.reason))
1102 except KeyboardInterrupt:
1103 return -11
1104 except:
1105 return -12
1106
1107##########
1108# PRINTOUT

Callers 1

fetch_filesFunction · 0.85

Calls 3

readMethod · 0.80
formatMethod · 0.80
closeMethod · 0.45

Tested by

no test coverage detected