Schema on dataframe





Providing schema while pulling the data from file is one of the small step to increase your databricks application performance. providing schema to the statement helps spark engine to know the data types of the fields in the file in advance and hence no need to do through the data to finalize it.














Comments

Post a Comment

Popular posts from this blog

Microsoft BI Implementation - Cube back up and restore using XMLA command

Databricks - incorrect header check