pgreaper 1.0.0 Documentation

PostgreSQL Default Settings

Before uploading to Postgres, you may want to configure the default connection settings. If default settings are provided, PGReaper can use these to create new databases so you won’t have to create them manually.

Uploading TXTs and CSVs to Postgres

• Loading TXTs and CSVs to PostgreSQL
  • API
  • Details
    • Loading Large Files
    • Schema Inference
    • Corrections PGReaper Can Make

Reading JSON

• Loading JSON
  • From Files
    • Flattening
    • API

Reading From Compressed (ZIP) Files

PGReaper is capable of copying specific files located in ZIP archives without decompressing them. (If you are interested in reading in GZIP, BZIP, or LZMA compressed files, use the compression parameter on the copy_csv() function.)

• Reading ZIP Archives
  • Step 1: Read the ZIP Archive
  • Step 2: Get the Specific File
  • Notes: File Opening Safety

The Table Data Structure

PGReaper contains a two-dimensional data structure creatively named Table. These are lightweight structures which are built on Python’s list containers but contain a lot of features for easily mapping them into SQL tables.

• Creating and Loading Table Objects
  • Motivation
    • Structure
    • Type-Inference
  • Brief Example: Creating and Loading Table Objects
  • Full Reference: Creating and Modifying Tables
    • Slicing
    • Exotic Methods
    • Operations Which Create New Tables
    • Groupby
    • Adding Rows to a Table
  • Full Reference: Loading Tables to Postgres
  • Dumping to Files
  • Jupyter Notebooks

pandas Integration

• Loading pandas DataFrames
• Benchmark/Example: pandas DataFrame COPY and UPSERT
  • COPY
    • Structure
    • Results
  • UPSERT
    • Results
    • Checking Our Work
  • Where’s the Bottleneck

HTML Parsing

pgreaper contains a rich HTML parsing module featuring automated <table> parsing and Jupyter notebook integration. Because it is a large module on its own, it has its own documentation page.

• HTML Table Parsing
  • Introduction
  • Step 1: Reading in HTML
    • a) Locally Saved HTML Files
    • b) From the Web
  • Step 2: Reviewing the Output
  • Step 3: Cleaning the Tables
  • Step 4: Saving the Results
  • PostgreSQL

Internals

Information for maintainers and forkers of pgreaper.

• PGReaper Internals: Schema Inference
  • ColumnList
• PGReaper Internals: Mappings

Index

• Index
• Module Index
• Search Page