Skip to main content

Posts

The best way to store metadata

Big data is all the rage, and as companies scramble to gather as much data as they possibly can, the eventual outcome is typically that you end up with more data than you know what to do with. So in this lake filled with files and objects, it's very easy for data to get lost. This is where metadata comes into play. In a simple way, metadata is additional information about your data, both automatically and manually created. For example, in the case of a picture, the automatic metadata can include the exact time the picture was taken, the type of camera that was used, the GPS location and the resolution, whereas some of the manual metadata can include the name of the author and a description. Having this type of metadata attached to every piece of data you store is crucial, especially as you scale up. One challenge that often presents itself is how to store this metadata, since most file formats don't allow you to add this metadata to the file itself. Using a database One common
Read more
Recent posts

Use case: Migrating an on-premise data lake to the cloud

  In this new series of blog posts called Use Cases , I'm going to go over specific projects that we've worked on with clients to address specific needs, how the project went, and the lessons retained from that experience. I won't go into specifics, but I'll cover the important bits. Hopefully this will be helpful if your organization is looking at undertaking a similar project. This week, I'll talk about a project where we had to migrate a large data lake from an on-premise Hadoop infrastructure to the cloud on AWS. Before any project can start, an analysis of the business needs and a design has to be constructed in order to decide what the solution will be. In this particular case, we weren't involved in the initial decision process, but the solution architect came up with a pretty good design. The company had around 30 TB of data sitting in an on-premise data lake. Data engineers and developers would use Hadoop clusters in order to run their ETL pipelines, an
Read more

Do you know where your NFT is stored?

  NFTs, or Non-Fungible Tokens , have become the hottest thing in crypto this year. You may have heard in the news about a digital art piece from the artist Beeple selling as an NFT for $69 million, or how the NBA has partnered with a company called Top Shot to sell digital card packs to people. The short definition is that an NFT is a special crypto token that proves ownership of a digital asset. It usually lives on the Ethereum blockchain, although other blockchains also support NFTs, and unlike a traditional real world item, ownership can be proven with complete certainty since the entire transaction is public and stored on the blockchain ledger. So whoever paid those crazy prices for these early releases can be safe to know that their investment will live for as long as the blockchain does, even though it's debatable whether they would ever find a buyer to pay more than they did. But one question that many people aren't taking into account is where these files actually li
Read more

When should you automate your work?

Repetition is the key to success in life, whether we're talking about exercising, doing crossword puzzles or other types of professional work, doing the same thing multiple times builds muscle memory. There certainly is a benefit to repeating the same task, whether you're trying to actually build muscles, or in the case of intellectual challenges, if you're trying to learn or improve your mental capacity. But there are tasks that don't really bring any benefit, they just take time. Those are mostly when you spend a lot of time to do menial tasks, things that don't really bring you any benefit. Whether you're doing IT tasks by clicking through UIs, typing commands in your computer, filling numbers in a spreadsheet, writing a report, or classifying papers, these tasks don't tend to bring you any benefit. This is when you should think about automating your work. As someone with a tech background, automation is something I default to very quickly. The good thing
Read more

Using the Binance Smart Chain with Python

For most people, crypto is just a way to invest in a highly speculative asset. Bitcoin is seen as digital gold, as something you can buy, hold and sell at a higher price. But crypto currencies and blockchain technologies are used in many more applications, and creating innovative products on top of these networks starts with connecting to the network. Whether you're trying to create a digital wallet, interact with smart contracts, or build a completely new application, you need to use code and connect to the blockchain of your choice. In this post, we'll use Python and connect to the Binance Smart Chain. While there is a lot of crypto documentation out there for JavaScript, the amount is smaller for Python, so hopefully this post will give you a good place to start. Connecting to the chain First, you'll need to download the web3 library. You can use pip to install it on your system: pip install web3 Once installed, open up a Python interpreter and let's initialize the
Read more

Running AWS commands using Jenkins pipelines

Jenkins is a well known tool used in the DevOps world to automate processes. Typically, it's mostly used to build and deploy software as part of a CI/CD process, including cloning code from git, running a build job, running tests and deploying the resulting artifacts. However, Jenkins pipelines can be used to automate anything at all. Because of its built-in credentials manager, multi-node architecture and multitude of plugins, it can be worth it to centralize a lot of the repeated tasks you may have inside of Jenkins, rather than having scripts resting on a server somewhere. In this post we'll see how to use a Jenkins pipeline to connect to AWS and retrieve a list of S3 buckets available. Storing credentials The first thing you should do is securely store your credentials. One of the oldest security issue that IT people have had to deal with is the management of credentials. If you use a shell script to run your command, then you need credentials to be stored inside the script
Read more

Crypto coins fundamentals

  As part of our technology services here at Dendory Capital, we touch a lot of exciting new industries. One of them is Cryptocurrencies. By now most people know about Bitcoin and how it's been going on a massive growth in the past year, but Bitcoin is just one of many different crypto coins. The landscape is much more vast than it used to be when Bitcoin was the only coin in town, and perhaps the most interesting part of crypto isn't so much the price action but instead all of the technologies behind these coins. In this post I will describe some of the basic fundamentals that drive crypto including blockchain networks, smart contracts and gas fees. The basics of blockchain All crypto currencies work on top of something called the blockchain. In simplified terms, a blockchain is a chain of blocks, linked together in a secure manner, and published on a ledger. This is how crypto coins are created, and how transactions are recorded. For example, let's say you have one Bitcoi
Read more