MCPcopy Create free account
hub / github.com/IBM/Project_CodeNet / tokenize.py

File tokenize.py

tools/tokenizer/tokenize.py:None–None  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1#!/usr/bin/env python
2
3# Copyright IBM Corporation 2021, 2022
4# Written by Geert Janssen <geert@us.ibm.com>

Callers

nothing calls this directly

Calls 2

tokenFunction · 0.85
decodeMethod · 0.80

Tested by

no test coverage detected