Issue With Df.show() In Pyspark

I have the following code: import pyspark import pandas as pd from pyspark.sql import SQLContext from pyspark.sql.functions import udf from pyspark.sql.types import IntegerType, S

Solution 1:

"Accept timed out" generally points to a problem with your spark instance. It may be overloaded or not enough resources (memory/cpu) to start your job or it might be a temporary network issue. You can monitor you jobs on Spark UI.

Also there is some issue with your code. You can use spark session object directly to create dataframe.

import pandas as pd

df_pd = pd.DataFrame(
    data={'integers': [1, 2, 3],
     'floats': [-1.0, 0.5, 2.7],
     'integer_arrays': [[1, 2], [3, 4, 5], [6, 7, 8, 9]]}
)
df = spark.createDataFrame(df_pd)
df.printSchema()