Python Channel

I'm loading many versions of JSON files to spark DataFrame. some of the files holds columns A,B… Read more Spark - Set Null When Column Not Exist In Dataframe

I tried taking a schema as a common schema by df.schema() and load all the CSV files to it .But fai… Read more Spark - Merge / Union Dataframe With Different Schema (column Names And Sequence) To A Dataframe With Master Common Schema

I am using Spark version 2.0.1 and trying to connect cloudant database using Python code but same t… Read more Cloudant Database Not Connecting Using Spark Python

I have a job requires to run on a partitioned spark dataframe, and the process looks like: rdd = sp… Read more Pyspark Convert Result Of Mappartitions To Spark Dataframe

I have a column (myCol) in a Spark dataframe that has values 1,2 and I want to create a new column … Read more Spark Udf With Dictionary Argument Fails

I am new to Python, so apologizing if there is any mistake in my approach I have a scenario where c… Read more In Pyspark Is There Any Way To Dynamically Register Udf Using Functions Of Python Class Given At Run Time?

I have three columns in a pyspark data frame ( sample data given below ) I wanted to get the remov… Read more Get 20th To 80th Percentile Of Each Group - Pyspark

I need to creeate an new Spark DF MapType Column based on the existing columns where column name is… Read more Pyspark: Create Maptype Column From Existing Columns

Is there any way to use custom record delimiters while reading a csv file in pyspark. In my file re… Read more Read A File In Pyspark With Custom Column And Record Delmiter

I've been able to import the package: import pixiedust pixiedust.installPackage('databricks… Read more How Do I Implement The Tensorframes Spark Package On Data Science Experience?

Let's say I have a PySpark data frame like so: 1 0 1 0 0 0 1 1 0 1 0 1 How can I append the la… Read more Pyspark - Append Previous And Next Row To Current Row

I applied an algorithm from the question Spark: How to transpose and explode columns with nested ar… Read more Spark: How To Transpose And Explode Columns With Dynamic Nested Arrays

I am unable to launch Pyspark in windows after installing Python 3.x with Anaconda. Getting below e… Read more Pyspark Launch Issue Windows 10 Wxith Python 3.6

In the spark, I understand how to use wholeTextFiles and textFiles, but I'm not sure which to u… Read more How Does The Number Of Partitions Affect `wholetextfiles` And `textfiles`?

I have Hortonworks HDP 2.6.3 running Spark2 (v2.2). My test case is very simple: Create a Hive tab… Read more Spark 2.2 Thrift Server Error On Dataframe Numberformatexception When Query Hive Table

My problem is based on the similar question here PySpark: Add a new column with a tuple created fro… Read more Create A Tuple Out Of Two Columns - Pyspark

I am trying to use data from a spark dataframe as the input for my k-means model. However I keep ge… Read more Pyspark - Valueerror: Could Not Convert String To Float / Invalid Literal For Float()

I have a PySpark DataFrame, df1, that looks like: CustomerID CustomerValue CustomerValue2 12 … Read more Cosine Similarity For Two Pyspark Dataframes

I defined a function in PySpark which is- def add_ids(X): schema_new = X.schema.add('id_col… Read more Ambiguous Behavior While Adding New Column To Structtype

I'm reading in some JSON on the from: {'a': [{'b': {'c': 1, 'd'… Read more Flatten Nested Array In Spark Dataframe