TikTok Scraper & Downloader

Scrape and download useful information from TikTok.

No login or password are required

This is not an official API support and etc. This is just a scraper that is using TikTok Web API to scrape media and related meta information.

---

---

Content

• Features
• To Do
• Contribution
• Installation
• Usage
  • In Terminal
    • Terminal Examples
    • Manage Download History
    • Scrape and Download in Batch
    • Output File Example
  • Docker
    • Build
    • Run
  • Module
    • Methods
    • Options
    • Use with Promises
    • Use with Events
    • How to get/set session value
    • How to access/download video
    • Output Example
      • Video Feed Methods
      • getUserProfileInfo
      • getHashtagInfo
      • getVideoMeta
      • getMusicInfo

Important notes

• As of right now it is NOT possible to download video without the watermark

Features

• Download unlimited post metadata from the User, Hashtag, Trends, or Music-Id pages
• Save post metadata to the JSON/CSV files
• Download media with and without the watermark and save to the ZIP file
• Download single video without the watermark from the CLI
• Sign URL to make custom request to the TikTok API
• Extract metadata from the User, Hashtag and Single Video pages
• Save previous progress and download only new videos that weren't downloaded before. This feature only works from the CLI and only if download flag is on.
• View and manage previously downloaded posts history in the CLI
• Scrape and download user, hashtag, music feeds and single videos specified in the file in batch mode

To Do

• [x] CLI: save progress to avoid downloading same videos
• [x] Rewrite everything in TypeScript
• [x] Improve proxy support
• [x] Add tests
• [x] Download video without the watermark
• [x] Indicate in the output file(csv/json) if the video was downloaded or not
• [x] Build and run from Docker
• [x] CLI: Scrape and download in batch
• [x] CLI: Load proxies from a file
• [x] CLI: Optional ZIP
• [x] Renew API
• [x] Set WebHook URL (CLI)
• [x] Add new method to collect music metadata
• [ ] Add Manual Pagination
• [ ] Improve documentation
• [ ] Download audio files
• [ ] Web interface

Contribution

• Don't forget about tests

yarn test

yarn build

Installation

tiktok-scraper requires Node.js v10+ to run.

Install from NPM

npm i -g tiktok-scraper

Install from YARN

yarn global add tiktok-scraper

USAGE

In Terminal

$ tiktok-scraper --help

Usage: tiktok-scraper <command> [options]

Commands:
  tiktok-scraper user [id]     Scrape videos from username. Enter only username
  tiktok-scraper hashtag [id]  Scrape videos from hashtag. Enter hashtag without #
  tiktok-scraper trend         Scrape posts from current trends
  tiktok-scraper music [id]    Scrape posts from a music id number
  tiktok-scraper video [id]    Download single video without the watermark
  tiktok-scraper history       View previous download history
  tiktok-scraper from-file [file] [async]  Scrape users, hashtags, music, videos mentioned
                                in a file. 1 value per 1 line

Options:
  --version            Show version number                             [boolean]
  --session            Set session cookie value. Sometimes session can be
                       helpful when scraping data from any method  [default: ""]
  --session-file       Set path to the file with list of active sessions. One
                       session per line!                           [default: ""]
  --timeout            Set timeout between requests. Timeout is in Milliseconds:
                       1000 mls = 1 s                               [default: 0]
  --number, -n         Number of posts to scrape. If you will set 0 then all
                       posts will be scraped                        [default: 0]
  --since              Scrape no posts published before this date (timestamp).
                       If set to 0 the filter is deactived          [default: 0]
  --proxy, -p          Set single proxy                            [default: ""]
  --proxy-file         Use proxies from a file. Scraper will use random proxies
                       from the file per each request. 1 line 1 proxy.
                                                                   [default: ""]
  --download, -d       Download video posts to the folder with the name input
                       [id]                           [boolean] [default: false]
  --asyncDownload, -a  Number of concurrent downloads               [default: 5]
  --hd                 Download video in HD. Video size will be x5-x10 times
                       larger and this will affect scraper execution speed. This
                       option only works in combination with -w flag
                                                      [boolean] [default: false]
  --zip, -z            ZIP all downloaded video posts [boolean] [default: false]
  --filepath           File path to save all output files.
      [default: "/Users/karl.wint/Documents/projects/javascript/tiktok-scraper"]
  --filetype, -t       Type of the output file where post information will be
                       saved. 'all' - save information about all posts to the`
                       'json' and 'csv'
                               [choices: "csv", "json", "all", ""] [default: ""]
  --filename, -f       Set custom filename for the output files    [default: ""]
  --noWaterMark, -w    Download video without the watermark. NOTE: With the
                       recent update you only need to use this option if you are
                       scraping Hashtag Feed. User/Trend/Music feeds will have
                       this url by default            [boolean] [default: false]
  --store, -s          Scraper will save the progress in the OS TMP or Custom
                       folder and in the future usage will only download new
                       videos avoiding duplicates     [boolean] [default: false]
  --historypath        Set custom path where history file/files will be stored
                   [default: "/var/folders/d5/fyh1_f2926q7c65g7skc0qh80000gn/T"]
  --remove, -r         Delete the history record by entering "TYPE:INPUT" or
                       "all" to clean all the history. For example: user:bob
                                                                   [default: ""]
  --webHookUrl         Set webhook url to receive scraper result as HTTP
                       requests. For example to your own API       [default: ""]
  --method             Receive data to your webhook url as POST or GET request
                                      [choices: "GET", "POST"] [default: "POST"]
  --help               Show help                                       [boolean]

Examples:
  tiktok-scraper user USERNAME -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper trend -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper hashtag HASHTAG_NAME -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper music MUSIC_ID -d -n 50 --session sid_tt=dae32131231
  tiktok-scraper video https://www.tiktok.com/@tiktok/video/6807491984882765062 -d
  tiktok-scraper history
  tiktok-scraper history -r user:bob
  tiktok-scraper history -r all
  tiktok-scraper from-file BATCH_FILE ASYNC_TASKS -d

• Terminal Examples
• Manage Download History
• Scrape and Download in Batch

Output File Example

Docker

By using docker you won't be able to use --filepath and --historypath , but you can set volume(host path where all files will be saved) by using -v

Build

docker build . -t tiktok-scraper

Run

Example 1: All files including history file will be saved in the directory(\$pwd) where you running the docker from

docker run -v $(pwd):/usr/app/files tiktok-scraper user tiktok -d -n 5 -s

Example 2: All files including history file will be saved in /User/blah/downloads

docker run -v /User/blah/downloads:/usr/app/files tiktok-scraper user tiktok -d -n 5 -s

Module

Methods

.user(id, options) //Scrape posts from a specific user (Promise)
.hashtag(id, options) //Scrape posts from hashtag section (Promise)
.trend('', options) // Scrape posts from a trends section (Promise)
.music(id, options) // Scrape posts by music id (Promise)

.userEvent(id, options) //Scrape posts from a specific user (Event)
.hashtagEvent(id, options) //Scrape posts from hashtag section (Event)
.trendEvent('', options) // Scrape posts from a trends section (Event)
.musicEvent(id, options) // Scrape posts by music id (Event)

.getUserProfileInfo('USERNAME', options) // Get user profile information
.getHashtagInfo('HASHTAG', options) // Get hashtag information
.signUrl('URL', options) // Get signature for the request
.getVideoMeta('WEB_VIDEO_URL', options) // Get video meta info, including video url without the watermark
.getMusicInfo('https://www.tiktok.com/music/original-sound-6801885499343571718', options) // Get music metadata

Options

const options = {
    // Number of posts to scrape: {int default: 20}
    number: 50,

    // Scrape posts published since this date: { int default: 0}
    since: 0,

    // Set session: {string[] default: ['']}
    // Authenticated session cookie value is required to scrape user/trending/music/hashtag feed
    // You can put here any number of sessions, each request will select random session from the list
    sessionList: ['sid_tt=21312213'],

    // Set proxy {string[] | string default: ''}
    // http proxy: 127.0.0.1:8080
    // socks proxy: socks5://127.0.0.1:8080
    // You can pass proxies as an array and scraper will randomly select a proxy from the array to execute the requests
    proxy: '',

    // Set to {true} to search by user id: {boolean default: false}
    by_user_id: false,

    // How many post should be downloaded asynchronously. Only if {download:true}: {int default: 5}
    asyncDownload: 5,

    // How many post should be scraped asynchronously: {int default: 3}
    // Current option will be applied only with current types: music and hashtag
    // With other types it is always 1 because every request response to the TikTok API is providing the "maxCursor" value
    // that is required to send the next request
    asyncScraping: 3,

    // File path where all files will be saved: {string default: 'CURRENT_DIR'}
    filepath: `CURRENT_DIR`,

    // Custom file name for the output files: {string default: ''}
    fileName: `CURRENT_DIR`,

    // Output with information can be saved to a CSV or JSON files: {string default: 'na'}
    // 'csv' to save in csv
    // 'json' to save in json
    // 'all' to save in json and csv
    // 'na' to skip this step
    filetype: `na`,

    // Set custom headers: user-agent, cookie and etc
    // NOTE: When you parse video feed or single video metadata then in return you will receive {headers} object
    // that was used to extract the information and in order to access and download video through received {videoUrl} value you need to use same headers
    headers: {
        'user-agent': "BLAH",
        referer: 'https://www.tiktok.com/',
        cookie: `tt_webid_v2=68dssds`,
    },

    // Download video without the watermark: {boolean default: false}
    // Set to true to download without the watermark
    // This option will affect the execution speed
    noWaterMark: false,

    // Create link to HD video: {boolean default: false}
    // This option will only work if {noWaterMark} is set to {true}
    hdVideo: false,

    // verifyFp is used to verify the request and avoid captcha
    // When you are using proxy then there are high chances that the request will be
    // blocked with captcha
    // You can set your own verifyFp value or default(hardcoded) will be used
    verifyFp: '',

    // Switch main host to Tiktok test enpoint.
    // When your requests are blocked by captcha you can try to use Tiktok test endpoints.
    useTestEndpoints: false
};