This should work. It has the advantage that you do not have to collect the result, as well as being more functional.
val messages = tweetDF.select("msg").as[String]
val positiveWords =
Source
.fromFile("/home/teslavm/positive.txt")
.getLines
.toList
.map(word => word.toLowerCase)
def hasPositiveWords(message: String): Boolean = {
val _message = message.toLowerCase
positiveWords.exists(word => _message.contains(word))
}
val positiveMessages = messages.filter(hasPositiveWords _)
println(positiveMessages.count())
I tested this code locally with:
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder.master("local[*]").getOrCreate()
import spark.implicits._
val tweetDF = List(
(1, "Yes I am happy"),
(2, "Sadness is a way of life"),
(3, "No, no, no, no, yes")
).toDF("id", "msg")
val positiveWords = List("yes", "happy")
And it worked.
