Big Data Is Out, Small Data Is In

The tech sector that serves real estate owners and operators, PropTech/CRETech or whatever your favorite term is, is a relatively young industry. The CRE industry has been served by software vendors since at least the 70’s when MRI was founded, but the idea of a cloud-based tech ecosystem didn’t really establish itself until the late 2000’s with PropTech 1.0. This wave of development was largely based around information portals like real estate listings. In the 2010’s, Proptech 2.0 emerged, which provided a wave of tools that gave users better decision making capabilities. The last couple of years has given way to the rise of Proptech 3.0 — tools that use of Big Data, AI/ML, AR/VR, IoT (internet of things) devices, GreenTech and host of other buzzwords.

The one I just can’t get over is Big Data. Platforms that leverage enormous datasets to provide quality information to help you run your business. Except, I have questions about that. Do they actually use big data? Should they use big data?

The truth is that, with few exceptions, no one in our industry uses Big Data, and honestly, I don’t think they should. I was recently invited onto the podcast When Real Estate Flows, to talk about this exact issue (Note: When Real Estate Flows is a production of my current employer). Let’s cover a few of the discussion points below.

What is Big Data?

Defining Big Data is difficult, and it changes frequently. To make it even more difficult to define, it’s not a fixed size. 1GB in 1960 would have been considered big data, now you can’t boot a computer without at least 4GB of RAM, let alone do anything with it. But it’s still not that much. Today, Big Data is incomprehensibly large. Commonly accepted definitions would put big data somewhere in the Petabytes (1K TB) or even Exabytes (1M TB) range.

The traditional measure of volume for Big Data is simply whether or not you can work with your data using standard database tools (i.e. can you store it and query it from a standard MySQL or Postgres database). It’s not really Big Data until you need special software or hardware in order to work with the data you’ve collected.

Most of you reading this right now can access every piece of data your company has ever produced about your properties on your laptop right now. For some of you, it might all be stored on your local hard drive. No matter how long you’ve been collecting that data, it’s not honestly not that much. Even if you have an entire terabyte of data about every property you’ve ever owned or even just underwritten, all of your data can fit on an iPhone 14. A single iPhone.

Think about that for a second. Everything you’ve ever produced fits on a single iPhone. If you’re extra vigilant about saving everything, maybe two. If your data can fit in my back pocket, it’s not big data.

If you’re building software to support your operations, then the solutions from Big Tech are useless to you. Your peers are companies like Stripe, Klarna, Plaid, Robinhood, Affirm, and if I might shamelessly plug my own employer for a moment — LoanBoss. Your problems are those of reliability and accuracy. You can safely disregard the solution of any company that has:

• Lots of data
• Lots of money
• Lots of leeway with security

Twitter can lose a tweet, Facebook can drop a like. Robinhood can’t lose a trade. Venmo can’t misplace a payment. You can’t sacrifice your data.

If you’re building software in PropTech, your problems are different than big tech problems. Having been there myself, I can tell you that your problems aren’t going to be in throughput — you’re not processing billions or trillions of transactions a second. You need something simple to operate (read: reliable), that can support encryption, and powered by roles-based controls.

It’s Not A Bad Thing

Big Data shouldn’t be something to aspire to as an end unto itself. Big data is difficult to wrangle into something useful, and the technologies needed to work with them aren’t as precise or reliable as those used on smaller data sets. In fact, at this year’s Gartner Data & Analytics Summit, one of the key takeaways was that big data is often a liability.

Stop collecting data just in case. Substitute big data with small data, and real data with synthetic data.

— Gartner Data & Analytics Summit 2022

You don’t need Big Data, you need Impactful Data. In order for that data to be impactful for 99% of what we do, it needs to be in a size appropriate for human comprehension, something that can address a specific problem. There’s actually a term for this — Small Data.

The CRE Industry has been practicing this without realizing it for years. If necessity is the mother of invention, then real estate is the matriarch. Commercial real estate has been underserved by the tech community for a long time, so firms have learned how to operate with the tools they have. Excel has been the cornerstone of the industry since its release, which can only process relatively small data sets.

Because of this limitation in processing power, CRE operators have gotten very good at identifying what data is important and what’s just noise. As a tech strategist, this means I can’t mask poor domain modelling and bad use cases behind a mountain of data. This is a good thing, and one that the CRE Tech industry as a whole should embrace.

Parting Thoughts

Be wary of the bleeding edge of tech, especially when buzz words are involved. Core business processes are best served by proven, reliable technologies, not the flashy and new. To be clear, this is in reference to the technology that powers a platform, not the platform itself (one is a technical risk, the other is a business risk). CRE is a small data industry, and the tools that manage it should reflect that reality.

If you liked this article, consider following me and subscribing to email updates whenever I post an article. You can also follow me on Twitter or connect with me on LinkedIn.