AWS Summit 2019: The Scout24 Data Platform: A Technical Deep Dive by Dr. Sean Gustafson & Raffael Dzikowski, Scout24 AG

Posted on Tue, 26. February 2019 in conference notes
#AWS #AWSSummit #AWSSummitBerlin #AWSSummit_2019 #Scout24 data platform

Immobilien scout24

  1. 500 employees
  2. 489 million euros revenue
  3. technical evolution: monolith, data warehouse, production db -> transformed to microservices

Goals: fast driven development ​ We think our Data Platform is a Product! similar to AWS and Salesforce, but smaller, hybrid model provided to customers. It’s not something that companies must use, they can decide on technologies, they provide one path. Inspired by DevOPs Handbook:

Guiding principle of the platform:

  • Autonomy for producers and consumers,
  • self service ETL,
  • self service data ingestion

Self Service Dat Ingestion

Accounts are divided into segments, they use Data Lake

​ Two ingestion options:

  1. Amazon Kinesis Data Firehouse and Kafka Connect
  2. Self-service ETL

DataWario - our wrapper of AWS Data Pipeline:

  • shortens development cycles
  • building scout24 ecosystem

We build also small Spark library for: - aggregating - snapshoting - filtering

Self-Service Analysis

  1. data analysis for different user groups
  2. consistent data management
  3. Datalake for scout24 toolset

​