How to count distinct?

Counting the number of distinct elements in a data set is a very common query. It can help give you an idea of how many duplicates you are dealing with. Let's say for example that you have a set of transactions, and you wish to detect if these transactions are either associated to a small set of frequent buying customers or performed by different customers. This can help you understand your clients and what type of marketing strategies you need to adopt.

Apache Mahout Samsara: The Quick Start

Last week the newest Apache Mahout 0.10 was released. One of the new features it has is a new math environment called “Samsara”, or Mahout Scala/Spark Bindings.

Samsara is a Linear Algebra library for Mahout. It’s written in Scala, which makes it possible to use operator overloading and it features nice R-like or Matlab-like syntax for basic Linear Algebra operations. For example, matrix multiplication is just X %*% Y. What is more, these operations can be distributed and run by an executing environment - currently by Apache Spark.

In this article we will see how to quickly set up a basic skeleton project and then we’ll try to do some very simple analysis on a 200 MB dataset.

Naive Bayes on Apache Flink

In this blog post we are going to implement a Naive Bayes classifier in Apache Flink. We are going to use it for text classification by applying it to the 20 Newsgroup dataset. To understand what is going on, you should be familiar with Java and know what MapReduce is. If you have seen and understood a word count example in any system, you're good to go. If you haven't heard of MapReduce or haven't seen the word count, you may first have a look at our introductory post "Hadoop and MapReduce".

Hadoop and MapReduce

In this article we will briefly discuss the computation paradigm MapReduce, and Apache Hadoop as one of its implementations. We won't get into much details, and we even won't implement the Word Count on Hadoop, but it should give some foundation for the future articles about tools for scalable data processing.

The Dark Side of Entrepreneurship

"We will have more than a million clients and our company will be top leader in the industry over the next year". This is what every first time entrepreneur says at some point in time.

We often hear stories about young entrepreneurs who dropped school at a very young age and had a huge success. We look at these very few success stories and, as entrepreneurs, we lie to ourselves that one day we will be like them...

You normally recognise entrepreneurs as those who change jobs very frequently. They try a bit of everything and, in the end, they don’t get deep into any of the topics. They like to taste a bit of everything. They change countries, jobs and friends and it seems that, everywhere they land, they find something to do. They are proactive and extremely curious. They just don’t find their place in any of the traditional companies. They are dreamers and born sellers, even if they have to sell things not even they can imagine.

Spring Batch Essentials

The book "Spring Batch Essentials" by Packt Publishing is in print now, and I had a pleasure to be a technical reviewer of this book.

Spring Batch is a tool for creating ETL ("Read/Process/Write") jobs: for batch processing large portions of enterprise data that requires sophisticated transformations and involves complex business logic. It gives you a possibility to manage jobs easily, supports transactions and allows job execution to be scaled to process large volumes of data.

Best Time to Learn Linear Algebra is Now!

Linear Algebra is a crucial prerequisite for many things, including Statistics, Data Mining, Machine Learning, Computer Vision, Image Processing and many many others, so it's very important to know the basics of Linear Algebra to understand more advanced concepts. For example, it's really helpful for our IT4BI studies, especially for the specialization at TU Berlin.

And the best time to learn Linear Algebra or refresh your knowledge about it is right now! At this moment there are a couple of nice MOOCs that have just started and a few more are about to start in the nearest future.

Even if you don't join right now, they should be available in the future for learning as self-paced versions. Additionally I would like to include my favorite video courses on Linear Algebra, they are also for learning at your own speed with no deadlines.