difference-between-stream-processing-and-message-processing

 difference-between-stream-processing-and-message-processing


Message Processing implies operations on and/or using individual messages. Stream Processing encompasses operations on and/or using individual messages as well as operations on collection of messages as they flow into the system. For e.g., let's say transactions are coming in for a payment instrument - stream processing can be used to continuously compute hourly average spend. In this case - a sliding window can be imposed on the stream which picks up messages within the hour and computes average on the amount. Such figures can then be used as inputs to fraud detection systems

Although Rabbit supports streaming, it was actually not built for it(see Rabbit´s web site) Rabbit is a Message broker and Kafka is a event streaming platform.

Kafka can handle a huge number of 'messages' towards Rabbit. Kafka is a log while Rabbit is a queue which means that if once consumed, Rabbit´s messages are not there anymore in case you need it.

However Rabbit can specify message priorities but Kafka doesn´t.

It depends on your needs.


Stream processing framework differs with input of data.In Batch processing,you have some files stored in file system and you want to continuously process that and store in some database. While in stream processing frameworks like Spark, Storm, etc will get continuous input from some sensor devices, api feed and kafka is used there to feed the streaming engine.


If you like splitting hairs: Messaging is communication between two or more processes or components whereas streaming is the passing of event log as they occur. Messages carry raw data whereas events contain information about the occurrence of and activity such as an order. So Kafka does both, messaging and streaming. A topic in Kafka can be raw messages or and event log that is normally retained for hours or days. Events can further be aggregated to more complex events.


Why is it called In-Stream?

The term “In-Stream Processing” means that a) the data is coming into the processing engine as a continuous “stream” of events produced by some outside system or systems, and b) the processing engine works so fast that all decisions are made without stopping the data stream and storing the information first.

You can think of an In-Stream Processing engine as a “cyber plant” for event processing. Imagine that events are coming in at high speeds to the front docks of the cyber plant where they are captured, sorted into queues, and sent on to the assembly lines for processing. Inside, on the cyber conveyor, specialized software robots perform analytical computations and transformations that filter, match, count, aggregate, and reason about the events as they are passed down the line. Whenever something interesting is discovered or computed — such as unmasking a fraud or computing a dynamic price — the notification is sent immediately to an external business system to do something about it. At the end of the conveyor, processed events are shipped to a warehouse where other systems can access them for other forms of processing.



Comments

Post a Comment

Popular posts from this blog

Read and Navigate XML - Beautiful Soup

 ### How to read and navigate XML There is a Python library called BeautifulSoup, which makes reading in and parsing XML data easier. Here is the link to the documentation: [Beautiful Soup Documentation](https://www.crummy.com/software/BeautifulSoup/) The find() method will find the first place where an xml element occurs. For example using find('record') will return the first record in the xml file: ```xml <record>   <field name="Country or Area" key="ABW">Aruba</field>   <field name="Item" key="SP.POP.TOTL">Population, total</field>   <field name="Year">1960</field>   <field name="Value">54211</field> </record> ``` The find_all() method returns all of the matching tags. So find_all('record') would return all of the elements with the `<record>` tag. Run the code cells below to get a basic idea of how to navigate XML with BeautifulSoup. To naviga...
Read more

WordNet in Python

Image
  WordNet   is an English dictionary which is a part of   Natural Language Tool Kit   (NLTK) for Python. This is an extensive library built to make   Natural Language Processing   (NLP) easy. Some basic functions will be discussed in this article. To start using WordNet, you have to import it first: from  nltk.corpus  import  wordnet Synsets and Lemmas In WordNet, similar words are grouped into a set known as a  Synset  (short for  Synonym-set ). Every Synset has a name, a part-of-speech, and a number. The words in a Synset are known as  Lemmas . Getting Synsets The function  wordnet.synsets('word')   returns an array  containing all the Synsets related to the word passed to it as the argument. Example: print ( wordnet.sysnets ( 'room' )) Output: [Synset(‘room.n.01’), Synset(‘room.n.02’), Synset(‘room.n.03’), Synset(‘room.n.04’), Synset(‘board.v.02’)] The method returned five Synsets; four have the name...
Read more