Skip to main content

Posts

Showing posts from November, 2020

Big Data analysis using Apache Spark

 The Dendory Capital Datalab creates solutions, workflows, tools and pipelines for our clients' Big Data needs. As part of this process, we need to make sure the solutions we provide our clients work properly. Today, we're going to go over a simple use case of analyzing a dataset to gain useful insights about a particular problem. We're going to load a CSV file containing data from the NASA Near Earth Objects project, and try to find out whether or not any large object is going to come close to the planet in the next week. Singing up for Databricks Databricks is the commercial version of Apache Spark, and provides a handy web-based interface to create and manage clusters, start a notebook, and use Python code without any administration overhead. Better yet, they have a community edition we're going to be able to use for free. So the first thing to do is go to databricks.com and signing up for a community account. Once you confirm your email address, you can log into

Notes taking software reviewed

Taking notes is something that anyone who works with intellectual matters has to do. You typically start in high school, when the things you learn are no longer trivial and can no longer be retained by heart just by hearing them, and it goes on throughout a lifetime for many of us. A lot of what makes a good engineer or scientist isn't what they can remember, but how good they are at finding the answer, whether that's by referring to your notes from previous problems you've solved, searching for the answer through experiments, or just Googling for it. If you want to retain that knowledge and not waste time when you need to do something that's close enough to something you've touched in the past, having a well organizing notes taking system is crucial. Over the years I've used a lot of tools myself, and I will review what I consider 3 of the best and most popular options here. Apple Notes If you use an iPhone, like many of us do, the default notes app that comes