Skip to content
milesgrangerPublic

Latest commit

 

History

19 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

pontem

Treat PySpark DataFrames like pandas.

This is currently just a hobby project, not suitable for use.

Turn somethinig like this:

# Pure PySpark API; df is type pyspark.sql.DataFrame
def multiply(n):
    return udf(lambda col: col * n, FloatType())
df = df.withColumn('new_col', df.select(multiply(2)(df['other_col'])))

...into this:

# Using pontem.core.DataFrame object.
df['new_col'] = df['other_col'] * 2

Releases

Packages

Contributors

Languages