Spark Context , Spark Session and JVM

 Why we have one spark context per JVM ?

handling single data intensive application is painful enough (tuning GC, dealing with leaking resources, communication overhead). Mutliple Spark applications running in a single JVM would be impossible to tune and manage in a long run.


If one of the processes hangs, or fails, or it's security is compromised, the others don't get affected.

I think having separate runtimes also helps GC, because it has less references to handle than if it was altogether.


What is JVM ?

The Java Virtual Machine (JVM) is the virtual machine that runs the Java bytecodes. The JVM doesn't understand Java source code; that's why you need compile your *.java files to obtain *.class files that contain the bytecodes understood by the JVM. It's also the entity that allows Java to be a "portable language" (write once, run anywhere). Indeed, there are specific implementations of the JVM for different systems (Windows, Linux, macOS, see the Wikipedia list), the aim is that with the same bytecodes they all give the same results


Spark Concepts 

how the resources are distributed when we create multiple spark session in one spark context


when multiple Spark sessions are created within a single Spark context, they share the same underlying resources allocated to the Spark context. The sessions allow for isolated SQL environments but do not have separate physical resources. The management of resources, including their allocation and potential dynamic adjustment, is handled at the Spark context level, affecting all sessions within the application.

In Apache Spark, SQL configurations are isolated across different Spark sessions to allow for different configurations and temporary views within the same Spark application. This isolation is achieved through the concept of session-scoped SQL configurations.


  • Separate Configurations: When you create a new SparkSession using the newSession() method on an existing SparkSession, the new session does not share SQL configurations with the parent session. This means that changes to configurations, such as those set using setConf method, in one session do not affect the other sessions.
  • Temporary Views: Temporary views and databases created in one SparkSession are scoped to that session and are not visible in other sessions. This allows users to create temporary tables with the same name in different sessions without conflict.
  • User-Defined Functions (UDFs): Similarly, UDFs registered in one SparkSession are not available in other sessions unless explicitly registered in each session.
  • Temporary views in Spark are session-scoped. This means that a temporary view created in one SparkSession is not visible in another SparkSession unless it's created as a global temporary view. Global temporary views are visible across all SparkSessions but are tied to a system-preserved database global_temp. This scoping allows for the same temporary view name to be reused across different sessions without conflict
https://medium.com/@kar9475/data-sharing-between-multiple-spark-jobs-in-databricks-308687c99897



Comments

Post a Comment

Popular posts from this blog

Read and Navigate XML - Beautiful Soup

 ### How to read and navigate XML There is a Python library called BeautifulSoup, which makes reading in and parsing XML data easier. Here is the link to the documentation: [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/) The find() method will find the first place where an xml element occurs. For example using find('record') will return the first record in the xml file: ```xml <record>   <field name="Country or Area" key="ABW">Aruba</field>   <field name="Item" key="SP.POP.TOTL">Population, total</field>   <field name="Year">1960</field>   <field name="Value">54211</field> </record> ``` The find_all() method returns all of the matching tags. So find_all('record') would return all of the elements with the `<record>` tag. Run the code cells below to get a basic idea of how to navigate XML with BeautifulSoup. To naviga...
Read more

difference-between-stream-processing-and-message-processing

 difference-between-stream-processing-and-message-processing Message Processing implies operations on and/or using  individual  messages. Stream Processing encompasses operations on and/or using individual messages as well as operations on collection of messages as they flow into the system. For e.g., let's say transactions are coming in for a payment instrument - stream processing can be used to continuously compute hourly average spend. In this case - a sliding window can be imposed on the stream which picks up messages within the hour and computes average on the amount. Such figures can then be used as inputs to fraud detection systems Although Rabbit supports streaming, it was actually not built for it(see Rabbit´s web site) Rabbit is a Message broker and Kafka is a event streaming platform. Kafka can handle a huge number of 'messages' towards Rabbit. Kafka is a log while Rabbit is a queue which means that if once consumed, Rabbit´s messages are not there anymore in c...
Read more

WordNet in Python

Image
  WordNet   is an English dictionary which is a part of   Natural Language Tool Kit   (NLTK) for Python. This is an extensive library built to make   Natural Language Processing   (NLP) easy. Some basic functions will be discussed in this article. To start using WordNet, you have to import it first: from  nltk.corpus  import  wordnet Synsets and Lemmas In WordNet, similar words are grouped into a set known as a  Synset  (short for  Synonym-set ). Every Synset has a name, a part-of-speech, and a number. The words in a Synset are known as  Lemmas . Getting Synsets The function  wordnet.synsets('word')   returns an array  containing all the Synsets related to the word passed to it as the argument. Example: print ( wordnet.sysnets ( 'room' )) Output: [Synset(‘room.n.01’), Synset(‘room.n.02’), Synset(‘room.n.03’), Synset(‘room.n.04’), Synset(‘board.v.02’)] The method returned five Synsets; four have the name...
Read more