I have csv file with about 5000 rows and 950 columns. First I load it to DataFrame: After that I search all string columns and want to index them. For that I create indexers for each string column and create pipeline But when I try to transform my initial dataframe with this pipeline I get StackOverflowError …
Tag: scala
How can I access values in a scala.collection.mutable.WrappedArray of WrappedArray’s in java
I am parsing a json file in SparkSQL in JAVA and I need to be able to access coordinates that are returned in what appears to be a WrappedArray of WrappedArrays. Here is the code: OUTPUT: WrappedArray(WrappedArray(30.74806, 40.79944)) file.json Answer Spark SQL Row has getList method, that returns a Java list…
Ways to make maven build faster?
I have a multi module java project. Maven takes almost around 40 secs to build it. I have tried maven with multi threaded builds too by specifying -T and -C args for no of threads and cores to be used. But I haven’t seen any significant improvement in wall time of my builds. I am using maven 3.2.3 and s…
Spark read file from S3 using sc.textFile (“s3n://…)
Trying to read a file located in S3 using spark-shell: The IOException: No FileSystem for scheme: s3n error occurred with: Spark 1.31 or 1.40 on dev machine (no Hadoop libs) Running from the Hortonworks Sandbox HDP v2.2.4 (Hadoop 2.60) which integrates Spark 1.2.1 out of the box Using s3:// or s3n:// scheme W…
How do I add a line break in my Twilio SMS message?
Using Java/Scala. I am sending this string (an sms message) to a user mobile number by a Twilio Account. This goes out to the user mobile, in 1 line. I want a newline after the first sentence. I want the user to receive: How can I enter a line break? Answer If you are using an official Twilio library, the
Difference between Java Optional and Scala Option
At the very end, this article introducing to new Java 8 Optional, states that Optional is not nearly as powerful as Option[T] in Scala (but at least it doesn’t allow wrapping null). The API is not as straightforward as null-handling and probably much slower. But the benefit of compile-time checking plus reada…
What is the advantage of using scala pattern matching instead of java switch case?
Everybody says that pattern matching is a great feature in functional languages. Why? Can’t I simple use ifs and switch cases for everything? I’d like to understand the advantages of using pattern matching instead of regular procedural programming ifs and switch cases Answer I’d first like t…
Creating a Java Enum in Scala
My workplace has been experimenting in moving from Java to Scala for some tasks, and it works well for what we’re doing. However, some preexisting logging methods expect a java.lang.Enum. The logging method is defined in the (Java) base class, and the subclasses can define their own enums, which the log…
Running jacocoReport
I’m using Gradle 1.7 and Jacoco plugin. My project uses Java and Scala plugins. When I run gradlew -i clean jacocoTestReport Report is not created and I see in the log What does it mean? Why report is not created? Answer The task will only run if coverage data is available. You can make sure of that by …
Maven package works but Intellij’s build fails
I have a JDK 1.7 project with a maven dependency to a local jar in my maven repo. I’m unable to build the project using Intellij, with the errors that a symbol cannot be found (the symbol is a class importing packages from the local jar) But I can successfully build the project using ‘mvn package&…