sroka package

Package providing simple Python access to data in: * Google AdManager (GAM earlier DoubleClick for Publishers, DFP) * Google Sheets * Google BigQuery * AWS Athena * AWS s3 * MySQL

Stale, no longer tested: * Google Analytics * MOAT * Qubole * Rubicon * Neo4j

Sroka library was checked to work for Python >=3.10, <=3.14.

Developers

Install requirements and enable custom githooks:

pip install -r requirements.txt
git config --local core.hooksPath .githooks/

Check style with flake8:

flake8 .

Please target Pull Requests against dev branch.

Installation

Pypi last release

pip install sroka

GitHub version (beta version)

pip install git+ssh://git@github.com/Wikia/sroka

Configuration

in home folder create ~/.sroka_config (hidden folder) file where you will store: * config.ini file based on config.sample.ini with information to access Qubole, MOAT, Athena, S3 and Rubicon * client_secrets.json for GA access * ad_manager.json for GAM access * credentials.json for Google sheets access * bigquery_credentials.json for BigQuery access

Alternatively, you may set localization of your files during analysis:

from sroka.config.config import setup_env_variables
from sroka.config.config import setup_client_secret
from sroka.config.config import setup_admanager_config
from sroka.config.config import setup_bigquery_config
from sroka.config.config import setup_google_sheets_credentials
setup_env_variables('/file_path/config.ini')
setup_client_secret('/file_path/client_secrets.json')
setup_admanager_config('/file_path/ad_manager.json')
setup_bigquery_config('/file_path/bigquery_credentials.json')
setup_google_sheets_credentials('/file_path/credentials.json')

Getting GA, GAM, BigQuery and Google docs jsons with secrets

Google Analytics

1. Use this wizard  to create or select a project in the Google Developers Console and automatically turn on the API. Click Continue, then Go to credentials.
2. On the Add credentials to your project page, click the Cancel button.
3. At the top of the page, select the OAuth consent screen tab. Select an Email address, enter a Product name if not already set, and click the Save button.
4. Select the Credentials tab, click the Create credentials button and select OAuth client ID.
5. Select the application type Other, enter the chosen name, and click the Create button.
6. Click OK to dismiss the resulting dialog.
7. Click the file_download (Download JSON) button to the right of the client ID.

GAM

1. Follow these instructions
   • while adding a service account note that the role needs to have necessary viewing and reporting permissions.

You should end up with .json (!) file with credentials

1. Make sure the Name in "OAuth 2.0 client IDs" matches the service account in "Service account keys": here
2. Create GAM account as service account not a new user: https://support.google.com/admanager/answer/6078734?hl=en
3. Once you have a service account, it can be used to access data in different networks. Simply add it as a new service account through GAM UI of the second network.
4. Additional information can be specified in config.ini file:
5. network code - a default value that can be overwritten in a function call
6. application name - custom name of your network, if not specified, a generic value will be passed.

Google drive sheets credentials

In order to authorize in Google Sheets you need to generate credentials in Google Console: * Create project and enable Sheets API * Configure OAuth consent screen * Add desktop credentials

You should end up with credentials.json file that should be downloaded to ~/.sroka_config folder.

Google BigQuery credentials

Go to link and follow up instructions within Setting up authentication section. You should end up with bigquery_credentials.json json file that should be downloaded to ~/.sroka_config folder.

Getting credentials & access tokens

Qubole

1. Find your Qubole API Token (go to user -> My Profile -> my_account -> API Token -> show)
2. Copy your Qubole API Token to config.ini file

Athena and s3 credentials

1. You should have your aws_access_key_id and aws_secret_access_key from registration process in AWS console.
2. s3bucket_name can be found in AWS console in Athena view when you click Settings, there you have Query result location. The name of location without s3:// and / is what you need.
3. For Athena usage you need to set also region (AWS regional endpoint), e.g. 'us-east-1'

Rubicon credentials

1. You should have your id, username and password from Rubicon
2. Copy values to config.ini file in relevant fields

MySQL connection information

1. In order to connect to a remote MySQL server, you need to provide the host and port values in the configuration. If it is accessible through a unix socket, you need to provide the path to this socket instead in the unix_socket configuration field.
2. If the MySQL server is protected by user credentials, you need to provide the user and password values in the configuration.
3. You can optionally specify the database to which you want to connect in the database configuration field.

Common issues

macOS

If you see an error like ValueError: unknown locale: UTF-8

Please add to ~/.bash_profile lines like this:

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

installing sroka

1. If PyYAML package is not building correctly, it may be caused by the fact that newer versions of pip won’t uninstall the package because it’s handled by disutils. Please install PyYAML package first with --ignore-installed flag.

2. If numpy gets messed up during sroka installation it is probably caused by multiple versions installed. Please uninstall all using pip uninstall and then reinstall latest one.

Google APIs cached files

If you encounter RefreshError similar to google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n "error": "invalid_grant",\n "error_description": "Bad Request"\n}') , try removing all files from ~/.cache directory.

We advise removing all files from ~/.cache directory for each new sroka update.

Credits

All people that contributed to sroka development before going opensource (including CR and QA): * martynaut * dorotamierzwa * fraszczakszymon * bckatarzyna * jacekbj * nandy-andy * dmnsobczak * szczeles * kvas-damian * pnather * philthyharry